Why Cloud Native Is Vital to Your Organization's APIs: The Impact Could Be More Than Expected
Top 10 Essential Linux Commands
Modern API Management
When assessing prominent topics across DZone — and the software engineering space more broadly — it simply felt incomplete to conduct research on the larger impacts of data and the cloud without talking about such a crucial component of modern software architectures: APIs. Communication is key in an era when applications and data capabilities are growing increasingly complex. Therefore, we set our sights on investigating the emerging ways in which data that would otherwise be isolated can better integrate with and work alongside other app components and across systems.For DZone's 2024 Modern API Management Trend Report, we focused our research specifically on APIs' growing influence across domains, prevalent paradigms and implementation techniques, security strategies, AI, and automation. Alongside observations from our original research, practicing tech professionals from the DZone Community contributed articles addressing key topics in the API space, including automated API generation via no and low code; communication architecture design among systems, APIs, and microservices; GraphQL vs. REST; and the role of APIs in the modern cloud-native landscape.
Open Source Migration Practices and Patterns
MongoDB Essentials
In modern web applications, integrating with external services is a common requirement. However, when interacting with these services, it's crucial to handle scenarios where responses might be delayed or fail to arrive. Spring Boot, with its extensive ecosystem, offers robust solutions to address such challenges. In this article, we'll explore how to implement timeouts using three popular approaches: RestClient, RestTemplate, and WebClient, all essential components in Spring Boot. 1. Timeout With RestTemplate First, let's demonstrate setting a timeout using RestTemplate, a synchronous HTTP client. Java import org.springframework.web.client.RestTemplate; import org.springframework.http.ResponseEntity; import org.springframework.http.HttpStatus; public class RestTemplateExample { public static void main(String[] args) { var restTemplate = new RestTemplate(); var url = "https://api.example.com/data"; var timeout = 5000; // Timeout in milliseconds restTemplate.getForEntity(url, String.class); System.out.println(response.getBody()); } } In this snippet, we're performing a GET request to `https://api.example.com/data`. However, we haven't set any timeout, which means the request might hang indefinitely in case of network issues or server unavailability. To set a timeout, we need to configure RestTemplate with an appropriate `ClientHttpRequestFactory`, such as `HttpComponentsClientHttpRequestFactory`. Java import org.springframework.web.client.RestTemplate; import org.springframework.http.client.HttpComponentsClientHttpRequestFactory; import org.springframework.http.ResponseEntity; import org.springframework.http.HttpStatus; public class RestTemplateTimeoutExample { public static void main(String[] args) { var url = "https://api.example.com/data"; var timeout = 5000; var clientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory(); clientHttpRequestFactory.setConnectTimeout(timeout); clientHttpRequestFactory.setConnectionRequestTimeout(timeout); var restTemplate = new RestTemplate(clientHttpRequestFactory); restTemplate.getForEntity(url, String.class); System.out.println(response.getBody()); } } 2. Timeout With WebClient WebClient is a non-blocking, reactive HTTP client introduced in Spring WebFlux. Let's see how we can use it with a timeout: Java import org.springframework.web.reactive.function.client.WebClient; import java.time.Duration; public class WebClientTimeoutExample { public static void main(String[] args) { var client = WebClient.builder() .baseUrl("https://api.example.com") .build(); client.get() .uri("/data") .retrieve() .bodyToMono(String.class) .timeout(Duration.ofMillis(5000)) .subscribe(System.out::println); } } Here, we're using WebClient to make a GET request to `/data` endpoint. The `timeout` operator specifies a maximum duration for the request to wait for a response. 3. Timeout With RestClient RestClient is a synchronous HTTP client that offers a modern, fluent API since Spring Boot 3.2. New Spring Boot applications should replace RestTemplate code with RestClient API. Now, let's implement a RestClient with timeout using `HttpComponentsClientHttpRequestFactory`: Java import org.springframework.http.client.HttpComponentsClientHttpRequestFactory; import org.springframework.web.client.RestTemplate; import org.springframework.web.reactive.function.client.WebClient; import java.time.Duration; public class RestClientTimeoutExample { public static void main(String[] args) { var factory = new HttpComponentsClientHttpRequestFactory(); factory.setConnectTimeout(5000); factory.setReadTimeout(5000); var restClient = RestClient .builder() .requestFactory(clientHttpRequestFactory) .build(); var response = restClient .get() .uri("https://api.example.com/data") .retrieve() .toEntity(String.class); System.out.println(response.getBody()); } } In this code, we define a specified timeout using HttpComponentsClientHttpRequestFactory and use it in RestClient.builder(). By setting timeouts appropriately, we ensure that our application remains responsive even in scenarios where external services are slow or unresponsive. This proactive approach enhances the overall reliability and resilience of our Spring Boot applications. Conclusion In summary, handling timeouts is important for web apps to stay responsive and robust during interactions with external services. We explored three popular Spring Boot approaches for implementing timeouts effectively: RestTemplate, WebClient, and RestClient. By setting appropriate timeouts, developers can ensure applications gracefully handle delayed or failed responses and enhance overall reliability and user experience in network conditions and service availability.
In many large organizations, software quality is primarily viewed as the responsibility of the testing team. When bugs slip through to production, or products fail to meet customer expectations, testers are the ones blamed. However, taking a closer look, quality — and likewise, failure — extends well beyond any one discipline. Quality is a responsibility shared across an organization. When quality issues arise, the root cause is rarely something testing alone could have prevented. Typically, there were breakdowns in communication, unrealistic deadlines, inadequate design specifications, insufficient training, or corporate governance policies that incentivized rushing. In other words, quality failures tend to stem from broader organizational and leadership failures. Scapegoating testers for systemic issues is counterproductive. It obscures the real problems and stands in the way of meaningful solutions to quality failings. Testing in Isolation In practice, all too often, testing teams still work in isolation from the rest of the product development lifecycle. They are brought in at the end, given limited information, and asked to validate someone else’s work. Under these conditions, their ability to prevent defects is severely constrained. For example, without access to product requirement documents, test cases may overlook critical functions that need validation. With short testing timelines, extensive test coverage further becomes impossible. Without insight into design decisions or access to developers, some defects found in testing prove impossible to diagnose effectively. Testers are often parachuted in when the time and cost of repairing a defect has grown to be unfeasible. In this isolated model, testing serves as little more than a final safety check before release. The burden of quality is passed almost entirely to the testers. When the inevitable bugs still slip through, testers then make for easy scapegoats. Who Owns Software Quality? In truth, responsibility for product quality is distributed across an organization. So, what can you do? Quality is everyone’s responsibility. Image sources: Kharnagy (Wikipedia), under CC BY-SA 4.0 license, combined with an image from Pixabay. Executives and leadership teams — Set the tone and policies around quality, balancing it appropriately against other priorities like cost and schedule. Meanwhile, provide the staffing, resources, and timescale needed for a mature testing effort. Product Managers — Gather user requirements, define expected functionality, and support test planning. Developers — Follow secure coding practices, perform unit testing, enable automated testing, and respond to defects uncovered in testing. User experience designers — Consider quality and testability during UX design. Conduct user acceptance testing on prototypes. Information security — Perform security reviews of code, architectures, and configurations. Guide testing-relevant security use cases. Testers — Develop test cases based on user stories, execute testing, log defects, perform regression test fixes, and report on quality to stakeholders. Operations — Monitor systems once deployed, gather production issues, and report data to inform future testing. Customers — Voice your true quality expectations, participate in UAT, and report real-world issues once launched. As this illustrates, no one functional area owns quality alone. Testers contribute essential verification, but quality is truly everyone’s responsibility. Governance Breakdowns Lead to Quality Failures In a 2023 episode of the "Why Didn’t You Test That?" podcast, Marcus Merrell, Huw Price, and I discussed how testing remains treated as a “janitorial” effort and cost center, and how you can align testing and quality. When organizations fail to acknowledge the shared ownership of software quality, governance issues arise that enable quality failures: Unrealistic deadlines — Attempting to achieve overly aggressive schedules often comes at the expense of quality and sufficient testing timelines. Leadership teams must balance market demands against release readiness. Insufficient investment — Success requires appropriate staffing and support for all areas that influence quality. These range from design and development to development to testing. Underinvestment leads to unhealthy tradeoffs. Lack of collaboration — Cross-functional coordination produces better quality than work done in silos. Governance policies should foster collaboration across product teams, not hinder it. Misaligned priorities — Leadership should incentivize balanced delivery, not just speed or cost savings. Quality cannot be someone else’s problem. Lack of transparency — Progress reporting should incorporate real metrics on quality. Burying or obscuring defects undermines governance. Absence of risk management — Identifying and mitigating quality risks through appropriate action requires focus from project leadership. Lacking transparency about risk prevents proper governance. When these governance breakdowns occur, quality suffers, and failures follow. However, the root causes trace back to organizational leadership and culture, not solely the testing function. The Costs of Obscuring Systemic Issues Blaming testers for failures caused by systemic organizational issues leads to significant costs: Loss of trust — When testers become scapegoats, it erodes credibility and trust in the testing function, inhibiting their ability to advocate for product quality. Staff turnover — Testing teams experience higher turnover when the broader organization fails to recognize their contributions and value. Less collaboration — Other groups avoid collaborating with testers perceived as bottlenecks or impediments rather than partners. Reinventing the wheel — Lessons from past governance breakdowns go unlearned, leading those issues to resurface in new forms down the line. Poorer customer experiences — Ultimately, obscuring governance issues around quality leads to more negative customer experiences that damage an organization’s reputation and bottom line. Taking Ownership of Software Quality Elevating quality as an organization-wide responsibility is essential for governance, transparency, and risk management. Quality cannot be the burden of one isolated function, and leadership should foster a culture that values quality intrinsically, rather than viewing it as an afterthought or checkbox. To build ownership, organizations need to shift testing upstream, integrating it earlier into requirements planning, design reviews, and development processes. It also requires modernizing the testing practice itself, utilizing the full range of innovation available: from test automation, shift-left testing, and service virtualization, to risk-based test case generation, modeling, and generative AI. With a shared understanding of who owns quality, governance policies can better balance competing demands around cost, schedule, capabilities, and release readiness. Testing insights will inform smarter tradeoffs, avoiding quality failures and the finger-pointing that today follows them. This future state reduces the likelihood of failures — but also acknowledges that some failures will still occur despite best efforts. In these cases, organizations must have a governance model to transparently identify root causes across teams, learn from them, and prevent recurrence. In a culture that values quality intrinsically, software testers earn their place as trusted advisors, rather than get relegated to fault-finders. They can provide oversight and validation of other teams’ work without fear of backlash. And their expertise will strengthen rather than threaten collaborative delivery. With shared ownership, quality ceases to be a “tester problem” at all. It becomes an organizational value that earns buy-in across functional areas. Leadership sets the tone for an understanding that if quality is everyone’s responsibility — so too is failure.
Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices. In the dynamic and rapidly evolving landscape of cloud-native and SaaS-driven software development, the process of API generation plays a pivotal role in accelerating time to market (TTM) and competitive advantage by rapidly creating APIs. This crucial function now converges with low- and no-code platforms, ushering in new ways of streamlined development processes. At the forefront of this evolution stands generative AI (GenAI), which offers unprecedented speed and flexibility in API creation. In this article, we embark on a journey to explore the transformative potential of generative AI and low- and no-code platforms in API generation by highlighting their roles in fostering innovation and expediting time to market. Additionally, we delve into strategies for effective implementation, compliance considerations, enterprise patterns, and the future trajectory of no-code API generation. Refer to Figure 1 for an illustration of the key components and architecture involved in this synergy: Generative AI – These algorithms autonomously generate code snippets by analyzing large datasets. Low-code/no-code platform – This encompasses the visual interface and pre-built components enabling API design without manual coding. API generation components – These leverage GenAI capabilities to automate API code generation based on user-defined specifications and prompts. API output – The final generated API code that is ready for deployment and integration into software applications. Generative vs. Conventional APIs in the Modern Software Landscape The debate between generative and conventional APIs has intensified, particularly concerning their impact on time to market and skill requirements. Generative APIs — powered by advanced technologies like artificial intelligence (AI) on top of low- and no-code platforms — have gained traction for their ability to automate the API generation process, promising faster TTM and reducing the skill threshold required for development. Conversely, conventional APIs — built through manual coding processes — typically demand a higher level of technical expertise and entail longer development cycles. In contrast, conventional APIs offer flexibility, tailored customization, and, hence, better capabilities for handling complexity. Table 1. Generative vs. conventional APIs Aspect Generative APIs Conventional APIs Time to market Rapid development cycles due to automation Lengthy development timelines Skill requirements Lower skill threshold; accessible to non-coders Require proficient coding skills Handling complex logic May struggle with replicating complex logic accurately Excel in handling complex logic and intricate business rules Integration complexity Simplified integration with existing systems May be challenging and time consuming Maintenance effort Reduced due to automation Require ongoing manual updates and maintenance Innovation potential Enable rapid experimentation and innovation May be limited due to manual coding and testing processes Error handling Automated but may be less customizable Fully customizable Legacy system integration May require additional effort Better compatibility and integration capabilities Creative freedom Limited due to automated processes Flexibility for developers Table 1 highlights the trade-offs between the two approaches, emphasizing the potential to revolutionize API development by expediting TTM and democratizing access to software development. Moreover, by lowering the skill threshold required for API development, these tools empower a broader range of individuals to contribute to the creation of APIs while developers can focus on higher-level tasks. The Synergy of Generative AI and Low and No Code for API Generation The integration of GenAI with low and no code for API generation signifies a significant advancement in API development/generation practices, particularly in terms of efficiency and accessibility. GenAI employs sophisticated large language model (LLM) algorithms that learn to autonomously analyze large datasets on enterprise (an organization's internal context) and open source patterns. When combined with low- and no-code platforms, GenAI enhances these tools by providing developers AI-generated code components that can be easily incorporated into their applications. This integration streamlines the development process, allowing developers to leverage pre-built AI-generated modules to accelerate prototyping and customization. Overall, the integration of GenAI with low- and no-code platforms revolutionizes API development, empowering organizations to innovate rapidly and deliver high-quality solutions to market with unprecedented speed and efficiency. Additionally, this amalgamation profoundly impacts the daily operations and responsibilities of engineers and developers, empowering them to innovate, streamline workflows, and adapt to the evolving demands of modern development practices. Figure 1. Architectural overview of GenAI with low- and no-code platforms for automated API generation Charting the Future: The Trajectory of No-Code API Generation As businesses prioritize agility and efficiency, the rising adoption of automated no-code API generation is set to transform development processes, therefore streamlining workflows and expediting TTM. With no-code platforms now able to accurately generate complex APIs by leveraging GenAI, this trajectory foresees a transition to more intuitive and potent development tools, which, in turn, will empower organizations to innovate swiftly and deliver top-notch solutions with unparalleled speed and efficiency. Democratizing Development As organizations adopt these innovative solutions, the democratization of development gains momentum. No-code API generation platforms empower individuals with diverse technical backgrounds to engage in API creation, promoting collaboration and inclusivity. By lowering entry barriers and reducing reliance on traditional coding skills, these platforms foster a future where API development is accessible and collaborative. This democratization accelerates innovation and ensures that a broader range of voices and perspectives contribute to the development process, resulting in more inclusive and impactful solutions. Compliance and Security In the domain of automated no-code API generation, organizations must prioritize compliance and security to uphold the integrity and reliability of their APIs. Compliance entails adhering to regulatory requirements such as GDPR, HIPAA, and PCI-DSS, as well as industry-specific benchmarks like ISO 27001 or SOC 2. Additionally, robust security measures — including authentication and encryption protocols — are essential for safeguarding against unauthorized access, secret rotations, and cyber threats. Prioritizing compliance and security enables organizations to mitigate risks, protect data, and uphold stakeholder trust. Business Transformation No-code API generation empowers businesses to swiftly adapt to market changes by facilitating rapid API creation and deployment without extensive coding. These tools and platforms enable developers to iterate quickly on ideas, prototypes, and solutions, expediting the development cycle and enabling organizations to respond promptly to market demands. By automating manual coding tasks and streamlining development processes, these tools free up developers' time and resources, allowing them to focus on more strategic tasks such as innovation, problem solving, and optimization. Additionally, these platforms democratize the development process by facilitating collaboration among cross-functional teams with varying levels of technical expertise, fostering scalability and competitiveness. Embracing no-code API generation is essential for driving meaningful transformation and maintaining a competitive edge in today's dynamic digital landscape. Challenges of Automated API Generation While these tools offer streamlined development processes, they come with hurdles that must be addressed. The following chart highlights key technical challenges, providing insights for effective implementation. Table 2. Challenges of automated API generation Challenge Description Handling complex data structures and business logic Handling complex data structures and business logic may pose challenges in integrating diverse data, validating data, and handling versioning effectively. Integration dependencies Automated tools may struggle to integrate seamlessly with existing systems and external APIs due to data format disparities, API versioning conflicts, and limited customization options. Security vulnerabilities Automated processes of low- and no-code results may be prone to potential vulnerabilities such as inadequate access controls, insecure configurations, and absence of data encryption, as well as vulnerabilities within third-party integrations. Limited flexibility Limited flexibility arises from predefined templates or patterns, constraining customization and adaptability to specific project requirements, potentially impacting functionality and scalability. Future proofing Low- and no-code API platforms are abstract in nature and can result in vendor lock-in; adaptability to evolving technologies as well as ensuring long-term support and compatibility can be challenging. Strategies and Guidelines for Generative No-Code API Development Strategies and guidelines provide a roadmap for leveraging AI-driven tools and low- and no-code platforms effectively. These strategies and guidelines encompass comprehensive planning, iterative development, and collaborative approaches, thus ensuring streamlined workflows and accelerated TTM while prioritizing automation, scalability, and security. Table 3. Developing generative no-code APIs Key Aspects Strategies and Guidelines Comprehensive planning Plan thoroughly by defining clear objectives and requirements up front to ensure alignment with business goals and user needs. Iterative development Adopt an iterative development approach, allowing for continuous feedback, testing, and refinement throughout the development process. Collaborative development Foster collaboration between technical and non-technical stakeholders, encouraging cross-functional teams to contribute to API design and development. Embrace automation Leverage automation tools and features provided by no- and low-code platforms to streamline development tasks and increase productivity. Ensure scalability Design APIs with scalability in mind, anticipating future growth and ensuring that the architecture can support increased demand and usage over time. Prioritize security Implement robust security measures to protect APIs from potential threats, including data breaches, unauthorized access, and injection attacks. Testing and validation Implement rigorous testing and validation processes to ensure the reliability, functionality, and interoperability of APIs across different platforms and environments. Conclusion As we cast our gaze toward the future shaped by cloud-native and SaaS-driven development, the integration of generative AI with low- and no-code platforms emerges as a catalyst for innovation. This symbiotic relationship not only revolutionizes API generation but also bestows developers with unprecedented flexibility and efficiency. Embracing automation and innovation will be pivotal in meeting the evolving market demands and expediting TTM. This trend represents more than just a leap in technological prowess; it signals a paradigm shift in the ethos of API development, where the context of creativity and efficiency converge harmoniously. Ultimately, developers and engineers are empowered by automated API generation tools, enabling them to rapidly translate ideas into prototypes and solutions, thus expediting the development cycle. This capability positions engineering and development teams to respond promptly to market demands and feature requirements, fostering experimentation and innovation. By automating manual coding tasks and streamlining development processes, these tools unlock opportunities for organizations to gain a competitive edge by delivering value to customers with unparalleled speed. Despite inevitable challenges, such as compliance and security considerations, the trajectory of automated API generation remains on a path of progress. Embracing strategic guidelines and proactively addressing challenges, businesses can harness the transformative potential of automated API generation to shape the future of software development and technology trends. This is an excerpt from DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices.Read the Free Report
The objective behind the solution described in this blog was to be able to share live 3D objects captured by one person, using “normal-looking” glasses, with another person who can then view them in XR (AR, VR, or MR) and/or 3D print them and do so with a similar experience as to what exists today for 2D pictures and 2D printers. About This Project While Meta Ray-Ban glasses are not XR headsets (they are smart glasses), they are currently the most unobtrusive, aesthetically “normal” looking glasses on the market that can be used to capture video (which can then be turned into 3D objects via an intermediary service) as they are indistinguishable from regular Ray-Ban Headliner and Wayfarer glasses aside from the small camera lenses that are mere millimeters in size. See the side-by-side comparison below. Ray-Ban Wayfarer glasses Meta Ray-Ban Wayfarer glasses The Oracle database plays a central role in the solution as it provides an optimized location for all types of data storage (including 3D objects and point clouds), various inbound and outbound API calls, and AI and spatial operations. Details can be found here. I will start by saying that this process will of course be more streamlined in the future as better hardware and software tech and APIs become available; however, the need for workflow logic, interaction with and exposure of APIs, and central storage will remain a consistent requirement of an optimal architecture for the functionality. It is possible to run both Java and JavaScript from within the Oracle database, load and use libraries for those languages, expose these programs as API endpoints, and make calls out from them. It is also possible to simply issue direct HTTP, REST, etc., commands from PL/SQL using the UTL_HTTP.BEGIN_REQUEST call or, for Oracle AI cloud services (or any Oracle cloud services), by using the DBMS_CLOUD.send_request call. This offers a powerful and flexible architecture where the following four combinations are possible. This being the case, there are several ways to go about the solution described here; for example, by issuing requests directly from the database or an intermediary external application (such as microservices deployed in a Kubernetes cluster) as shown in the previous diagrams. Flow The flow is as follows: The user takes a video with Ray-Bans. The video is automatically sent to Instagram (or Cloud Storage). The Oracle database calls Instagram to get the video and saves it in the database (or object storage, etc.). The Oracle database sends video to the photogrammetry/AI service and retrieves the 3D model/capture generated by it. Optionally, further spatial and AI operations are automatically conducted on the 3D model by the Oracle database. Optionally, further manual modifications are made to the 3D model and a manual workflow step may be added (for example, to gate 3D printing). From here the 3D capture/model can be 3D printed or viewed and interacted with via an XR (VR, AR, MR) headset - or both can be done in parallel. 3D Printing The Oracle database sends the 3D model (.obj file) to PrusaSlicer, which generates and returns G-code from it. The G-code print job is then sent to 3D printer via OctoPrint API server. XR Viewing and Interaction The 3D model is exposed as REST (ORDS) endpoint. The XR headset (Magic Leap 2, Vision Pro, Quest, etc.) receives the 3D model from Oracle database and renders it for viewing and interaction at runtime. In diagram form, the flow looks roughly like this: Now let’s elaborate on each step. Step 1: The User Takes a Video With Ray-Bans As mentioned earlier, I did not pick Meta Ray-Bans due to their XR functionality. Numerous other glasses have actual XR functionality well beyond Ray-Bans, full-on XR headsets with an increasingly better ability to do 3D scans of various types, and 3D scanners devoted to extremely accurate, high-resolution scans. I picked Ray-Bans because they are the glasses that are, in short, the most “normal” looking (without thick lenses or bridges that sit far from the face or extra extensions, etc.). Meta Ray-Bans have a “hey Meta” command that works like Alexa or Siri, though fairly limited at this point, as it can not refer to location services, can send but not read messages, etc. It’s not hard to see how it is possible to use Vision AI, etc. with them. However, built-in functionality does not exist currently, and more importantly, there is no API to access any functionality (there are access hacks out there but this blog will stick to legit, supported approaches), so it is limited for developers at this point. It can play music and, most importantly, take pictures and videos — that is the functionality I am using here. Streaming must be set up on the Meta View app and the Instagram account being streamed to must be a business account. However, both of these are simple configuration steps that can be found in the documentation and do not require additional cost. Step 2: Video Is Automatically Sent to Instagram (Or Cloud Storage) Ray-Ban video recording is limited to one-minute clips, but that is enough for any modern photogrammetry/AI engine to generate a 3D model of small to medium objects. Video taken with the glasses can be stored in cloud services such as iCloud and Google. However, it is not automatically synced until the glasses are placed in the glass case. This is why I opted for storage in Instagram reels, which, not surprisingly, is supported by the Meta Ray-Bans such that videos can be automatically streamed and saved there as they are taken. Setup steps to stream can be found here. 3. Oracle Database Calls Instagram (App) to Get the Video Here, the Oracle database itself listens/polls for new Instagram reels/videos, using the Instagram Graph API. This requires creating a Meta/Instagram application, etc., and here are the steps involved in doing so. Register as a Meta developer. Create an app and submit it for approval. This takes approximately 5 days if everything has been completed correctly for eligibility. This process, in particular for the Instagram Basic Display app type we are creating, is described on this Getting Started page. However, I will provide a few additional screenshots here to elaborate a bit as certain new items around app types, privileges, and app approval processes are missing from the doc. First, it is necessary to select a Consumer app type and then the Instagram Basic Display product. Then the app is submitted for approval for the instagram_graph_user_profile and instagram_graph_user_media: Finally, testers are added/invited for access tokens to be generated. Once the application is set up and access token(s) acquired, a list of the information about the media from the account is obtained by issuing a request in the following format: https://graph.instagram.com/{{IG_USER_ID}/media?fields=id,caption,media_url,media_type,timestamp,children{media_url}&limit=10&access_token={{IG_LONG_LIVED_ACCESS_TOKEN} Finally, the media desired is filtered out from that JSON returned (i.e., any new videos posted), and the media_url can be used to get the actual media. As stated before, the video can be retrieved from the URL using PL/SQL, JavaScript, or Java from within the Oracle database itself, or via an intermediary service called from the database. It can then be saved in the database, object storage, or other storage and sent to the photogrammetry/AI service. An example of doing this with JavaScript from inside the database can be found in my blog How To Call Cohere and Hugging Face AI From Within an Oracle Database Using JavaScript and an example using PL/SQL is presented here: PLSQL CREATE TABLE file_storage ( id NUMBER PRIMARY KEY, filename VARCHAR2(255), file_content BLOB ); DECLARE l_http_request UTL_HTTP.req; l_http_response UTL_HTTP.resp; l_blob BLOB; l_buffer RAW(32767); l_amount BINARY_INTEGER := 32767; l_pos INTEGER := 1; l_url VARCHAR2(1000) := 'https://somesite.com/somefile.obj'; BEGIN INSERT INTO file_storage (id, filename, file_content) VALUES (1, 'file.obj', EMPTY_BLOB()) RETURNING file_content INTO l_blob; -- Open HTTP request to download file l_http_request := UTL_HTTP.begin_request(l_url); UTL_HTTP.set_header(l_http_request, 'User-Agent', 'Mozilla/4.0'); l_http_response := UTL_HTTP.get_response(l_http_request); -- Download the file and write it to the BLOB LOOP BEGIN UTL_HTTP.read_raw(l_http_response, l_buffer, l_amount); DBMS_LOB.writeappend(l_blob, l_amount, l_buffer); l_pos := l_pos + l_amount; EXCEPTION WHEN UTL_HTTP.end_of_body THEN EXIT; END; END LOOP; UTL_HTTP.end_response(l_http_response); COMMIT; DBMS_OUTPUT.put_line('File downloaded and saved.'); EXCEPTION WHEN UTL_HTTP.end_of_body THEN UTL_HTTP.end_response(l_http_response); WHEN OTHERS THEN UTL_HTTP.end_response(l_http_response); RAISE; END; This same technique can be used for any call out from the database and storing of any file/content. Therefore, these snippets can be referred back to for saving any file including the .obj file(s) generated in the next step. Step 4: Oracle Database Sends Video to the Photogrammetry/AI Service and Retrieves the 3D Model/Capture Generated by It There are a few photogrammetry/AI (and Nerf, Splat, etc.) services available. I have chosen to use Luma Labs again because it has an API available for direct HTTPS calls, and examples are also given for over 20 programming languages and platforms. The reference for it can be found here. I will keep things short by giving the succinct curl command for each call in the flow, but the same can be done using PL/SQL, JavaScript, etc. from the database as described earlier. Once a Luma Labs account is created and luma-api-key created, the process of converting the video to a 3D .obj file is as follows: Create/initiate a capture. Shell curl --location 'https://webapp.engineeringlumalabs.com/api/v2/capture' \ --header 'Authorization: luma-api-key={key}' \ --data-urlencode 'title=hand' # example response # { # "signedUrls": { # "source": "https://storage.googleapis.com/..." # }, # "capture": { # "title": "hand", # "type": "reconstruction", # "location": null, # "privacy": "private", # "date": "2024-03-26T15:54:08.268Z", # "username": "paulparkinson", # "status": "uploading", # "slug": "pods-of-kon-66" # } # } This call will return a signedUrls.source URL that is then used to upload the video. Also note the generated slug value returned which will be used to trigger 3D processing, check status processing status, etc. Shell curl --location --request PUT 'https://storage.googleapis.com/...' \ --header 'Content-Type: text/plain' \ --data 'hand.mov' Once the video file is uploaded, the processing is triggered by issuing a POST request to the slug retrieved in step 1. Shell curl --location -g --request POST 'https://webapp.engineeringlumalabs.com/api/v2/capture/{slug}' \ --header 'Authorization: luma-api-key={key}' If the process is triggered successfully, a value of true will be returned and the following can be issued to check the status of the capture by calling the capture endpoint. Shell curl --location -g 'https://webapp.engineeringlumalabs.com/api/v2/capture/{slug}' \ --header 'Authorization: luma-api-key={key}' Once the status returned is equal to complete, the 3D capture zip file (which contains the .obj file as well as the .mtl material mapping file and .png texture files) is downloaded and saved by calling the download endpoint. The approaches mentioned earlier can be used to do this and save the file(s). Step 5: Optionally, Further Spatial and AI Operations Are Automatically Conducted on the 3D Model by the Oracle Database It is also possible to break down the .obj file and store its various vertices, vertice texture/material mappings, etc. in a table as a point cloud for analysis and manipulation. Here is a simple example of that: PLSQL create or replace procedure gen_table_from_obj(id number) as ord MDSYS.SDO_ORDINATE_ARRAY; f UTL_FILE.FILE_TYPE; s VARCHAR2(2000); i number; begin ord := MDSYS.SDO_ORDINATE_ARRAY(); ord.extend(3); f := UTL_FILE.FOPEN('ADMIN_DIR','OBJFROMPHOTOAI.obj', 'R'); i := 1; while true loop UTL_FILE.GET_LINE(f, s); if(s = '') then exit; end if; if(REGEXP_SUBSTR(s, '[^ ]*', 1, 1) = 'v') then ord(1) := TO_NUMBER(REGEXP_SUBSTR(s, '[^ ]*', 1, 3)); ord(2) := TO_NUMBER(REGEXP_SUBSTR(s, '[^ ]*', 1, 5)); ord(3) := TO_NUMBER(REGEXP_SUBSTR(s, '[^ ]*', 1, 7)); insert into INP_OBJFROMPHOTOAI_TABLE(val_d1, val_d2, val_d3) values (ord(1), ord(2), ord(3)); end if; i := i+1; end loop; end; / The Oracle database has had a spatial component for decades now, and recent versions have added several operations for different analyses of point clouds, mesh creation, .obj export, etc. These are described in this video. One operation that has existed for a number of releases is the pc_simplify function shown below. This is often referred to as "decimate" or other terms by various 3D modeling tools and provides the ability to reduce the number of polygons in a mesh, thus reducing overall size. This is handy for a number of reasons, such as when different clients will use the 3D model: for example, a phone with limited bandwidth or need for high-poly meshes. PLSQL procedure pc_simplify( pc_table varchar2, pc_column varchar2, id_column varchar2, id varchar2, result_table_name varchar2, tol number, query_geom mdsys.sdo_geometry default null, pc_intensity_column varchar2 default null) DETERMINISTIC PARALLEL_ENABLE; Step 6: Optionally, Further Manual Modifications Can Also Be Made to the 3D Model, and a Manual Approval Can Be Inserted as Part of the Workflow The 3D model can be loaded from the database, and edited in 3D modeling tools like Blender or 3D printing tools such as BambuLabs numerous others. Due to the many steps involved in this overall process, the solution is also a good fit for a workflow engine such as the one that exists as part of the Oracle database. In this case, a manual review/approval can be inserted as part of the workflow to prevent sending or printing undesired models. From here the 3D capture/model can be 3D printed or viewed and interacted with via an XR (VR, AR, MR) headset - or both can be done in parallel. 3D Printing Step 1: Oracle Database Sends the 3D Model (.Obj File) to PrusaSlicer Which Generates and Returns G-Code From It PrusaSlicer is an extremely robust and successful open-source project/application that takes 3D models (.stl, .obj, .amf) and converts them into G-code instructions for FFF printers or PNG layers for mSLA 3D printers. It supports every conceivable printer and format; however, does not provide an API, only a CLI. There are a few ways to work/hack around this for automation. One, shown here, is to implement a [Spring Boot] microservice that takes the .obj file and executes the PrusaSlicer CLI (which must be accessible to the microservice of course), returning the G-code. Java @RestController public class SlicerController { @PostMapping(value = "/slice", consumes = MediaType.MULTIPART_FORM_DATA_VALUE) public byte[] sliceStlFile(@RequestParam("file") MultipartFile file, @RequestParam("config") String configPath) throws IOException, InterruptedException { Path tempDir = Paths.get(System.getProperty("java.io.tmpdir")); Path stlFilePath = Files.createTempFile(tempDir, "stl", ".stl"); file.transferTo(stlFilePath.toFile()); Path gcodePath = Files.createTempFile(tempDir, "output", ".gcode"); // Execute PrusaSlicer CLI command String command = String.format("PrusaSlicer --slice --load %s --output %s %s", configPath, gcodePath.toString(), stlFilePath.toString()); Process prusaSlicerProcess = Runtime.getRuntime().exec(command); prusaSlicerProcess.waitFor(); // Return G-code byte[] gcodeBytes = FileUtils.readFileToByteArray(gcodePath.toFile()); Files.delete(stlFilePath); Files.delete(gcodePath); return gcodeBytes; } } Once the G-Code for the .obj file has been returned, it can be passed to the OctoPrint API server for printing. Step 2: G-Code Print Job Is Then Sent to 3D Printer via Octoprint API Server OctoPrint is an application for 3D printers that offers a web interface for printer control. It can be installed on essentially any computer (in the case of my setup, a minimal Raspberry Pi) that is connected to the printer. This can even be done over Wi-Fi, cloud, etc. depending on the printer and setup. However, we will keep it to this basic setup. Again, printers have different applications to provide this functionality, but OctoPrint provides a REST API, which allows for programmatic control, including uploading and printing G-code files. First, an API key must be obtained from OctoPrint’s web interface under Settings > API. Then, the G-code file is uploaded and immediately printed by using a call of this format/content: Shell curl -k -X POST "http://octopi.local/api/files/local" \ -H "X-Api-Key: API_KEY" \ -F "file=@/path/to/file.gcode" \ -F "select=true" \ -F "print=true" XR Viewing and Interaction Step 1: 3D Model Is Exposed as a REST (ORDS) Endpoint Any data stored in the Oracle Database can be exposed as a REST endpoint. To expose the .obj file for download by the XR headset, we can REST-enable the table created earlier using the following: PLSQL CREATE OR REPLACE PROCEDURE download_file(p_id IN NUMBER) IS l_blob BLOB; BEGIN SELECT file_content INTO l_blob FROM file_storage WHERE id = p_id; -- Use ORDS to deliver the BLOB to the client ORDS.enable_download(l_blob); EXCEPTION WHEN NO_DATA_FOUND THEN HTP.p('File not found.'); WHEN OTHERS THEN HTP.p('Error retrieving file.'); END download_file; This makes the file (i.e., the .obj, etc. files) accessible via a simple GET call. Shell curl -X GET "http://thedbserver/ords/theschema/file/file/{id}" -o "3Dmodelwithobjandtextures.zip" Here, we are exposing and downloading the file by id. The XR headset keeps track of the 3D models it has received and polls for the next id each time. From here, the 3D model can be viewed on computers, phones, etc. as-is. However, it is obviously more interactive to view in an actual XR headset, which is what I will describe next. Step 2: XR Headset (Magic Leap 2, Vision Pro, Quest, etc.) Receives the 3D Model From the Oracle Database and Renders It For Viewing and Interaction at Runtime The process for receiving and rendering 3D in Unity as I am showing here (and likewise for UnReal) is the same regardless of the headset used. Interaction with 3D objects (via e.g., hand tracking, eye gaze, voice, etc.) has also been standardized with OpenXR and WebXR which Magic Leap, Meta, and others are compliant with. However, Apple (similar to the case of phone, etc. development) has its own development SDK, ecosystem, etc., and regardless, interaction is not the crux of this blog, so I will only cover the important aspects of the 3D object for viewing. There are a couple of assets on the Unity asset store for doing this conveniently. What is shown below is greatly simplified, but explains the general approach. First, the 3D model is downloaded using a script like this: C# using System.Collections; using UnityEngine; using UnityEngine.Networking; using System.IO; public class DownloadAndSave3DMOdel : MonoBehaviour { private string ordsFileUrl = "http://thedbserver/ords/theschema/file/file/{id}"; private string filePathForTextures; void Start() { filePath = Path.Combine(Application.persistentDataPath, "3Dmodelwithobjandtextures.zip"); StartCoroutine(DownloadFile(ordsFileUrl)); } IEnumerator DownloadFile(string url) { using (UnityWebRequest webRequest = UnityWebRequest.Get(url)) { yield return webRequest.SendWebRequest(); if (webRequest.isNetworkError || webRequest.isHttpError) { Debug.LogError("Error: " + webRequest.error); } else { //Unpack the zip here and process each file for the case where it isObjFile, isMtlFile, or isTextureFile; //The texture/png files are written to a file like this if (isTextureFile) File.WriteAllBytes(filePathForTextures, webRequest.downloadHandler.data); //Whereas the .obj and .mtl files are converted to a stream like this else if (isObjFile){ var memoryStream = new MemoryStream(Encoding.UTF8.GetBytes(webRequest.downloadHandler.text)); processObjFile(memoryStream); } else if (isMtlFile){ var memoryStream = new MemoryStream(Encoding.UTF8.GetBytes(webRequest.downloadHandler.text)); processMtlFile(memoryStream); } } } } } As shown, the the texture/.png files in the zip are saved and the .obj and .mtl files are converted to MemoryStreams for processing/creating the Unity GameObject's MeshRenderer, etc. This parsing is similar to how we created a point cloud table from the .obj in the earlier optional step where we parse the lines of the .obj file; however, a bit more complicated as we also parse the .mtl file (spec explaining these formats can be found here) and apply the textures to create the end resultant 3D model that is rendered for viewing at runtime. This entails this basic logic which will create the GameObject that can then be placed in the headset wearer's FOV or part of a library menu, etc. to interact with. C# // Create the Unity GameObject that will be the main result/holder of our 3D object var gameObject = new GameObject(_name); // add a MeshRenderer and MeshFilter to it var meshRenderer = gameObject.AddComponent<MeshRenderer>(); var meshFilter = gameObject.AddComponent<MeshFilter>(); // Create a Unity Vector object for each of the "v"/Vertices, "vn"/Normals, and "vt"/UVs values parsed from each of the lines in the .obj file new Vector3(x, y, z); // Create a Unity Mesh and add all of the Vertices, Normals, and UVs to it. var mesh = new Mesh(); mesh.SetVertices(vertices); mesh.SetNormals(normals); mesh.SetUVs(0, uvs); // Similarly parse the .mtl file to create the array of Unity Materials. var material = new Material(); material.SetTexture(...); material.SetColor(...); //collect materials into materialArray // Add the mesh and/with materials meshRenderer.sharedMaterials = materialArray; meshFilter.sharedMesh = mesh; Conclusion As you can see, there are many steps to the process; however, hopefully, you have found it interesting to see how it is possible — and will only become easier — to share 3D objects the way we share 2D pictures today. Please let me know if you have any thoughts or questions whatsoever and thank you very much for reading! Source Code The source code for the project can be found here. Video
Before We Start, What’s a Chapter? A chapter is a craft community (sometimes also referred to as a Practice or CoE) whose primary purpose is to help practitioners of the same craft or discipline (e.g., development, testing, business analysis, scrum mastery, UX, etc.) elevate their mastery of that particular craft or discipline. Chapters are also typically tasked with setting the standards and guidelines for performing that craft in the organization. Credit: Ashley-Christian Hardy, “Agile Team Organisation: Squads, Chapters, Tribes and Guilds,” 2016 TL/DR In Agile organizations, chapters (Chapters, Practices, CoEs, etc.) pursue the systematic development of capability and craft. This pursuit adds a lot of value to the organization (better quality, better knowledge retention, higher employee engagement, etc.). Chapters often struggle to pinpoint where they need to improve or how they can add more value to their members and the organization. Many organizations don’t offer clear guidelines to chapters (and chapter leads) as to what exactly is expected of them or what good looks like. This article presents a simple diagnostic (click to download the diagnostic) that a chapter could use to identify areas where they must focus their improvement efforts. It defines what ‘good’ looks like in the context of a chapter and provides a tool to help the chapter assess where they stand against that definition (where they’re doing well and where they need to improve). In the second part of this series, I will share several experiments that could be run to optimize each dimension of the chapter's effectiveness. I will also share a case study of how this model was implemented at scale at a large organization. Key Terms First, let’s define some of the key terms that will be used throughout this article: Craft refers to the specific discipline, domain, or skill set around which the chapter is formed. e.g., QA is a craft; UX is a craft; business analysis is a craft, etc. A craftsperson is a practitioner of a craft (developer, QA specialist, marketer, business analyst, etc.) I use the term performing the craft to refer to the actual day-to-day carrying out of tasks by a craftsperson (chapter member) – e.g. a QA specialist performs their craft by carrying out QA tasks. Craft quality (quality of the craft) refers to how well the craft is being performed. I sometimes use craftsmanship, which refers to craft mastery and excellence. Knowledge base refers to a centralized repository or system where craft-related information, best practices, documentation, standards, etc. are stored and shared among chapter members (and others across the organization). Standards (craft standards) refer to the established guidelines and principles that define the expected level of quality within a specific craft. Learning journey refers to the ongoing formal and informal learning efforts (training programs, hands-on application of new knowledge, mentoring, etc.) intended to extend existing skills and build new skills, and how that learning is expected to be acquired over time. Is It Worth Reading This Article? Well, if any of the following statements resonate with you, then I would strongly suggest that you read on: As an organization (or tribe, business unit, etc.): “We have a high risk of losing critical knowledge if a small number of our people leave” “We struggle with onboarding and, despite hiring top talent, it’s difficult for them to hit the ground running” “Despite hiring people with a lot of experience to mentor and grow our junior staff, we feel that knowledge sharing isn’t happening” “We invest a lot in training our people – sending them to courses, etc. – but we don’t feel that investment has had much impact on the overall quality of our products” “We have knowledge siloes even within the same discipline — there are islands of expertise that aren’t connected” “We fear that when the contractors and external consultants leave, our people won’t be able to deliver the same level of high craft quality” “Team members struggle when moving from one team to another due to a lack of consistency in how things are done” “Our team members seem to struggle to tap into the collective expertise of the team, leading to a lot of re-inventing the wheel and time wasted” While these are all difficult problems that result from complex interactions of causes that affect and are affected by every aspect of how the organization works, these issues are all heavily influenced by how effective we are at developing craftsmanship — that is, how good we are at developing craft mastery and excellence. Given that in Agile organizations craft communities (chapters, practices, CoEs, etc.) are the primary custodians of developing craftsmanship, what I’m proposing here is that by systematically assessing and optimizing the effectiveness of how these craft communities work, we can make great strides at addressing these challenges. So, Why Care About Chapters? Effective chapters create the conditions that enable high product quality, high employee satisfaction, and low knowledge loss risk. This is because effective chapters are good at developing master craftspeople. People who feel mastery of their craft are typically happier and more engaged, their output and the speed at which that output is delivered is superior, and, due to the fact that there’s more than a small number of them (and that there’s a robust process to develop more), the departure of a few won’t be catastrophic for the organization. Members of an effective chapter (that is, a chapter that’s good at developing the craftsmanship of its members), would typically say things like: Our chapter follows a systematic approach to building our capability and craft mastery (defining what capability needs to be built, providing its members with the mechanisms to plan how to build those capabilities, and providing the resources and support needed to implement that plan) Our chapter has in place the process and systems that enable us to leverage and build upon the accumulated formal knowledge that the chapter has amassed over time – the standards, playbooks, guidelines, best practices, lessons learned, etc. Our chapter has nurtured a rich social network that enables us to tap into the collective informal (tacit) knowledge and expertise of the chapter – the knowledge, nuances, and highly contextual experiences that aren’t documented anywhere (most knowledge is tacit) Our chapter follows a systematic approach to measuring the impact (outcomes) of craftsmanship-building and capability uplift efforts and leveraging the feedback to guide further craftsmanship-building efforts If we improve the effectiveness of a chapter (that is, optimize the 4 factors mentioned above that are key leading indicators to chapter effectiveness), we would increase the chapter’s ability to develop its members into craftspeople, which will positively affect and improve problems such as high knowledge loss risk, siloes, ineffective knowledge sharing, and low product quality. How Do We Improve Chapter Effectiveness? The first step to improving chapter effectiveness is to systematically assess how the chapter is performing against the 4 key dimensions of chapter effectiveness identified above (access to documented (formal) knowledge; systematic capability building; access to tacit (informal) knowledge; and systematic craft quality measurement and continuous improvement). In this 2-part series, I will introduce a simple diagnostic tool to assess chapter effectiveness (Part 1 – this article), and then delve into how to use the insights from the assessment to identify areas of improvement and how to go about setting chapter effectiveness goals and planning, implementing, and learning from chapter effectiveness improvement actions (Part 2). How Do We Measure Chapter Effectiveness? In this section, we will first go over the dimensions comprising it in some detail, and then present the diagnostic tool itself. Chapter Effectiveness Dimensions Dimension #1 The comprehensiveness, fitness-for-purpose, and ease of access (and use) of our craft standards and knowledge base – this is a broad area that covers things like how good we are at leveraging (and creating/documenting) existing knowledge, the ease of access to relevant knowledge, the accuracy and relevance of the knowledge chapter members can draw from (and its alignment with industry best practices), and the effective distilment of ‘lessons learned,’ which represents how outside knowledge is contextualized to fit the unique context of the organization, among other factors. Dimension #2 The effectiveness of the chapter’s effort to uplift the capability and craftsmanship of its members — effective chapters are good at describing what mastery of their craft means (what skills to acquire, what the levels of mastery of each skill look like, etc.), helping chapter members pinpoint where they are on that journey, and then collaborate as a team to envision what the path to mastery looks like for each member. They’re also good at taking those plans and turning them into reality: not only providing the resources and mentorship, but also the encouragement and peer support, keeping each other accountable, and measuring the outcomes of their elevated levels of craft mastery. Dimension #3 The effectiveness of tacit (informal) knowledge sharing between chapter members – Effective chapters realize that most knowledge is tacit – that is, not documented anywhere. Tacit knowledge is difficult to extract or express, and therefore difficult to formally document or write down. How do we effectively leverage knowledge that isn’t documented? By nurturing a thriving social network that allows chapter members to feel comfortable reaching out to each other for help, seek advice, ask questions, share interesting insights, etc. Such a network doesn’t just happen – it requires conscious, persistent effort to build. The statements comprising this dimension seek to explore some of the leading indicators to building effective knowledge-sharing and advice-seeking social networks. Dimension #4 The effectiveness of the chapter’s efforts to systematically and continuously improve craft quality – how do we know if the actions we’re undertaking to uplift the quality of our craft (committing to learning and capability uplift, fostering stronger knowledge-sharing networks, etc.) are delivering value? How do we know if the investment we’re putting into uplifting our capability into specific tools or frameworks is generating the returns expected? Effective chapters are really good at measuring and evaluating the quality of their craft across the organization (quantitatively and/or qualitatively). They’re good at setting SMART craft improvement goals given their understanding of how well the craft is being implemented and where they need to focus and invest in improvement, planning how to implement those goals, and good at implementing those plans (and learning from that implementation). This is a significant challenge area for many chapters, as it is often difficult to ‘measure’ the quality of how the craft is being implemented. The Chapter Effectiveness Diagnostic Introduction The diagnostic (click here to download the pdf version) comprises a number of statements that are intended to capture what ‘good’ looks like for that particular dimension. Chapter members are expected to rate how well they believe each statement describes the reality of their chapter on a scale ranging from 'completely disagree' to 'completely agree.' All chapter members (including their chapter lead) should take part in completing this diagnostic. One option to fill it (what many chapters do) is to send it out as a survey first, then discuss results or insights in one or more follow-up workshops. The purpose of this diagnostic is to serve as a conversation starter. As with all diagnostic and maturity models, the questions are merely intended to prompt us to have a discussion. The comments, anecdotes, and insights chapter members share as the team moves from one item to another provide a wealth of information. That information is what’s going to guide us (as a chapter) as we attempt to optimize the outcomes our chapter is creating. There’s no particular magic to this (or any) assessment model – it simply provides a structure within which good conversations can take place. What’s in the Pack? This pack contains the statements comprising this diagnostic model. Next to each statement is a brief explanation of why having a conversation about that statement is important and what to look for (and how to dig deeper and uncover insights) if the score against that particular statement is low. In the appendix, you'll find a printable version of the diagnostic (a template with only the statements), which can be distributed as handouts. Next Steps If you want to run the diagnostic as a survey, copy the statements into your survey tool. You may set the response options for each statement as completely disagree — disagree — neutral — agree — completely agree. Alternatively, you might opt for a sliding scale of 1-5, for example, or use whatever rating method you prefer to enable your team to assess how well each statement describes its reality. OK, We Ran the Diagnostic – What’s Next? As mentioned earlier, the conversation that follows this self-assessment is where we really get the value. As a team, the chapter gets together to reflect, explore, and try to make sense of the results of their chapter effectiveness self-assessment. They reflect on where they seem to be doing well and where they’re struggling, where they seem to all have the same experience, and where the scores reflect a difference in their perceptions. They reflect on common themes, outliers, relationships, and connections between statements, explore why some statements are not correlated even though they were expected to (and vice versa), and any other interesting insights that came out of the assessment. In the second part of this series, we will do a deep dive into how to translate these insights and learning into experiments and actions and measure the impact they create in optimizing chapter effectiveness. We will explore how to prioritize chapter effectiveness interventions, what experiments to run to uplift each chapter effectiveness dimension, and how to establish a robust continuous improvement cycle to consistently and systematically seek higher chapter effectiveness. We will go through a case study from a large financial organization where this model was applied at scale across a large number of chapters, and share some of the learnings from that experience.
With video surveillance increasingly becoming a top application of smart technology, video streaming protocols are getting a lot more attention. We’ve recently spent a lot of time on our blog posts discussing real-time communication, both to and from video devices, and that has finally led to an examination of the Real-Time Streaming Protocol (RTSP) and its place in the Internet of Things (IoT). What Is the Real-Time Streaming Protocol? The Real-Time Streaming Protocol is a network control convention that’s designed for use in entertainment and communications systems to establish and control media streaming sessions. RTSP is how you will play, record, and pause media in real time. Basically, it acts like the digital form of the remote control you use on your TV at home. We can trace the origins of RTSP back to 1996 when a collaborative effort between RealNetworks, Netscape, and Columbia University developed it with the intent to create a standardized protocol for controlling streaming media over the Internet. These groups designed the protocol to be compatible with existing network protocols, such as HTTP, but with a focus specifically on the control aspects of streaming media, which HTTP did not adequately address at the time. The Internet Engineering Task Force (IETF) officially published RTSP in April of 1998. Since the inception of RTSP, IoT developers have used it for various applications, including for streaming media over the Internet, in IP surveillance cameras, and in any other systems that require real-time delivery of streaming content. It’s important to note that RTSP does not actually transport the streaming data itself; rather, it controls the connection and the streaming, often working in conjunction with other protocols like the Real-time Transport Protocol (RTP) for the transport of the actual media data. RTSP works on a client-server architecture, in which a software or media player – called the client – sends requests to a second party, i.e., the server. In an IoT interaction, the way this works is typically that the client software is on your smartphone or your computer and you are sending commands to a smart video camera or other smart device that acts as the server. The server will respond to requests by performing a specific action, like playing or pausing a media stream or starting a recording. And you’ll be able to choose what the device does in real-time. Understanding RTSP Requests So, the client in an RTSP connection sends requests. But what exactly does that mean? Basically, the setup process for streaming via RTSP involves a media player or feed monitoring platform on your computer or smartphone sending a request to the camera’s URL to establish a connection. This is done using the “SETUP” command for setting up the streaming session and the “PLAY” command to start the stream. The camera then responds by providing session details so the RTP protocol can send the media data, including details about which transport protocol it will use. Once the camera receives the “PLAY” command through RTSP, it begins to stream packets of video data in real-time via RTP, possibly through a TCP tunnel (more on this later). The media player or monitoring software then receives and decodes these video data packets into viewable video. Here’s a more thorough list of additional requests and their meanings in RTSP: OPTIONS: Queries the server for the supported commands. It’s used to request the available options or capabilities of a server. DESCRIBE: Requests a description of a media resource, typically in SDP (Session Description Protocol) format, which includes details about the media content, codecs, and transport information. SETUP: Initializes the session and establishes a media transport, specifying how the media streams should be sent. This command also prepares the server for streaming by allocating necessary resources. PLAY: Starts the streaming of the media. It tells the server to start sending data over the transport protocol defined in the SETUP command. PAUSE: Temporarily halts the stream without tearing down the session, allowing it to be resumed later with another PLAY command. TEARDOWN: Ends the session and stops the media stream, freeing up the server resources. This command effectively closes the connection. GET_PARAMETER: Used to query the current state or value of a parameter on the session or media stream. SET_PARAMETER: Allows the client to change or set the value of a parameter on the session or media stream. Once a request goes through, the server can offer a response. For example, a “200 OK” response indicates a successful completion of the request, while “401 Unauthorized” indicates that the server needs more authentication. And “404 Not Found” means the specified resource does not exist. If that looks familiar, it’s because you’ve probably seen 404 errors and a message like “Web page not found” at least once in the course of navigating the internet. The Real-Time Transport Protocol As I said earlier, RTSP doesn’t directly transmit the video stream. Instead, developers use the protocol in conjunction with a transport protocol. The most common is the Real-time Transport Protocol (RTP). RTP delivers audio and video over networks from the server to the client so you can, for example, view the feed from a surveillance camera on your phone. The protocol is widely used in streaming media systems and video conferencing to transmit real-time data, such as audio, video, or simulation data. Some of the key characteristics of RTP include: Payload type identification: RTP headers include a payload type field, which allows receivers to interpret the format of the data, such as the codec being used. Sequence numbering: Each RTP data packet is assigned a sequence number. This helps the receiver detect data loss and reorder packets that arrive out of sequence. Timestamping: RTP packets carry timestamp information to enable the receiver to reconstruct the timing of the media stream, maintaining the correct pacing of audio and video playback. RTP and RTSP are still not enough on their own to handle all the various tasks involved in streaming video data. Typically, a streaming session will also involve the Real-time Transport Control Protocol (RTCP), which provides feedback on the quality of the data distribution, including statistics and information about participants in the streaming session. Finally, RTP itself does not provide any mechanism for ensuring timely delivery or protecting against data loss; instead, it relies on underlying network protocols such as the User Datagram Protocol (UDP) or Transport Control Protocol (TCP) to handle data transmission. To put it all together, RTP puts data in packets and transports it via UDP or TCP, while RTCP helps with quality control and RTSP only comes in to set up the stream and act like a remote control. RTSP via TCP Tunneling While I said you can use both UDP and TCP to deliver a media stream, I usually recommend RTSP over TCP, specifically using TCP tunneling. Basically, TCP tunneling makes it easier for RTSP commands to get through network firewalls and Network Address Translation (NAT) systems. The reason this is necessary is because RTSP in its out-of-box version has certain deficiencies when it comes to authentication and privacy. Basically, its features were not built for the internet of today which is blocked by firewalls on all sides. Rather than being made for devices on local home networks behind NAT systems, RTSP was originally designed more for streaming data from central services. For that reason, it struggles to get through firewalls or locate and access cameras behind those firewalls, which limits its possible applications. However, using TCP tunneling allows RTSP to get through firewalls and enables easy NAT traversal while maintaining strong authentication. It allows you to use an existing protocol and just “package” it in TCP for enhanced functionality. The tunnel can wrap RTSP communication inside a NAT traversal layer to get through the firewall. This is important because it can be difficult to set up a media stream between devices that are on different networks: for example, if you’re trying to monitor your home surveillance system while you’re on vacation. Another benefit of TCP tunneling is enhanced security. Whereas RTSP and RTP don’t have the out-of-box security features of some other protocols, like WebRTC, you can fully encrypt all data that goes through the TCP tunnel. These important factors have made RTSP via TCP tunneling a top option for video streaming within IoT. Final Thoughts In summary, while RTSP provides a standardized way to control media streaming sessions, its inherent limitations make it challenging for modern IoT video use cases requiring remote access and robust security. However, by leveraging TCP tunneling techniques, developers can harness the benefits of RTSP while overcoming firewall traversal and encryption hurdles. As video streaming continues to drive IoT innovation, solutions like RTSP over TCP tunneling will be crucial for enabling secure, real-time connectivity across distributed devices and networks. With the right protocols and services in place, IoT developers can seamlessly integrate live video capabilities into their products.
Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices. REST APIs have become the standard for communication between applications on the web. Based on simple yet powerful principles, REST APIs offer a standardized yet flexible approach to the design, development, and consumption of programming interfaces. By adopting a client-server architecture and making appropriate use of HTTP methods, REST APIs enable smooth, efficient integration of distributed systems. Becoming a standard, the API ecosystem has grown much richer in recent years and is increasingly integrated into the DevOps ecosystem. It has been infused with agility, CI/CD, and FinOps, and continues to develop by itself. In this article, we're going to compile these new practices and tools to give you a large overview of what an "API approach" can do. API Design and Documentation The API design and documentation stage is crucial, as it defines the basis for all subsequent development. For this reason, it is essential to use methodologies such as domain-driven design (DDD), event storming, and API goals canvas — which represents what, who, how, inputs, outputs, and goals — to understand business needs and identify relevant domains and the objectives of the APIs to be developed. These workshops enable businesses and dev teams to work together and define API objectives and interactions between different business domains. Figure 1. Designing APIs When designing and documenting APIs, it's essential to take into account the fundamental principles of REST APIs. This includes identifying resources and representing them using meaningful URLs, making appropriate use of HTTP methods for CRUD (Create, Read, Update, Delete) operations, and managing resource states in a stateless way. By adopting a resource-oriented approach, development teams can design REST APIs that are intuitive and easy to use for client developers. REST API documentation should highlight available endpoints, supported methods, accepted and returned data formats, and any security or pagination constraints. As such, the REST principles do not exclude the freedom to make a number of choices, such as the choice of naming conventions. For a compilation of these best practices, you can read my last Refcard, API Integration Patterns. In this stage, API style books play a crucial role. It gives design guidelines and standards to ensure the consistency and quality of the APIs developed. These style books define rules on aspects such as URI structure, HTTP methods to be used, data formats, error handling, and so on. They serve as a common reference for all teams working on API projects within the organization. Stoplight and SwaggerHub are commonly used, but a simple Wiki tool could be enough. Data model libraries complete this phase by providing reusable data models, which define the standard data structures used in APIs. Data model libraries include JSON schemas, database definitions, object models, and more. They facilitate development by providing ready-to-use assets, reducing errors, and speeding up development. Commonly used tools include Apicurio and Stoplight. A workflow API's description is often missing from the APIs we discover on API portals. Questions arise such as: How do I chain API calls? How do I describe the sequence of calls? With a drawing? With text in the API description? How do I make it readable and regularly updated by the person who knows the API best (i.e., the developer)? It could still be a pain to understand the sequence of API calls. However, this is often covered by the additional documentation that can be provided on an API portal. Yet at the same time, this was decoupled from the code supplied by the developers. The OpenAPI Specification allows you to define links and callbacks, but it is quickly limited to explaining things properly. This is why the OpenAPI Workflows Specification has recently appeared, allowing API workflows to be defined. In this specification, the steps are always described in JSON, which, in turn, allows a schema to be generated to explain the sequence of calls. If you want to describe your workflows from OpenAPI specifications, you can use Swagger Editor or SwaggerHub. And you can use Swagger to UML or openapi-to-plantuml. If you want to begin by designing sequence diagrams, you can use PlantUml or LucidChart, for instance. There is no unique toolchain that fits all needs; you have to first know if you prefer a top-down or bottom-up approach. Tools such as Stoplight Studio combined with Redocly are commonly known for handling these topics — Apicurio as well. They can be used for API design, enabling teams to easily create and visualize OpenAPI specifications using a user-friendly interface. These specifications can then be used to automatically generate interactive documentation, ensuring that documentation is always up to date and consistent with API specifications. API Development Once the API specifications have been defined, the next step is to develop the APIs following the guidelines and models established during the design phase. Agile software development methods, effective collaboration, and version management are must-have practices to ensure good and fast developments. Figure 2. Building APIs For versioning, teams use version control systems such as Git or GitHub to manage API source code. Version control enables seamless collaboration between developers and ensures full traceability of API changes over time. During development, the quality of the API specification can be checked using linting tools. These tools can check: Syntax and structure of the code Compliance with coding standards and naming conventions Correct use of libraries and frameworks Presence of dead or redundant code Potential security problems Swagger-Lint and Apicurio Studio or Stoplight can be used to carry out these and other linting checks, but these checks can be made into a CI/CD toolchain (more info to come in the API Lifecycle Management section). Automation plays a crucial role in this stage, enabling unit, security, and load tests to run seamlessly throughout the development process. Postman and Newman are often used to automate API testing to ensure quality and security requirements, but other solutions exist like REST Assured, Karate Labs, and K6. Development frameworks supporting API REST development are very common, and the most popular ones include Express.js with Node.js, Spring Boot, and Meteor. Most of the popular frameworks support HTTP, so it should not be a complicated action to choose. API capacities are a must when you choose a framework, but they are not the only ones. Developers will build your stack, so you'll need frameworks that are both appreciated by your devs and relevant to other technical challenges you'll have to tackle. And we have to speak about mock prototyping! It's something that could unlock developers' inter-dependency to propose a Mock API whenever you target internal or external developers. This is generally based on the OpenAPI description of your API and is often taken into account by API management portals. There are also dedicated OSS projects such as MockServer or WireMock. API Security API security is a major concern in API development and management. It is essential to implement authentication, authorization, and data encryption mechanisms to protect APIs against attacks and privacy violations. API keys, OAuth 2.0, and OpenID Connect are the three protocols to know: API keys are still widely used for API access due to their ease and low overhead. They are a unique set of characters sent as a pair, a user, and a secret, and should be stored securely like passwords. OAuth 2.0 is a token-based authentication method involving three actors: the user, the integrating application (typically your API gateway), and the target application. The user grants the application access to the service provider through an exchange of tokens via the OAuth endpoint. OAuth is preferred for its granular access control and time-based limits. OpenID Connect is a standardization of OAuth 2.0 that adds normalized third-party identification and user identity. It is recommended for fine-grained authorization controls and managing multiple identity providers, though not all API providers require it. In addition to that, solutions such as Keycloak can be deployed to provide centralized management of identity and API access. Alternatives of Keycloak include OAuth2 Proxy, Gluu Server, WSO2 Identity Server, and Apache Syncope. But just talking about tools and protocols would not be enough to cover API security. Contrary to what we sometimes read, a front-end web application firewall (WAF) implementing the OWASP rules will prevent many problems. And what certainly requires a dedicated DZone Refcard, like Getting Started With DevSecOps, a comprehensive DevSecOps approach will greatly reduce the risks. However, automated security testing is also essential to guarantee API robustness against attacks. OSS tools such as ZAP can be used to tackle automated security tests, identifying potential vulnerabilities in APIs and enabling them to be corrected before they can be exploited by attackers. API Lifecycle Management Once APIs have been developed, they need to be deployed and managed efficiently throughout their lifecycle. This involves version management, deployment management, performance monitoring, and ensuring API availability and reliability. API management platforms include, but are not limited to, Gravitee, Tyk, WSO2 API Manager, Google Cloud Apigee, and Amazon API Gateway are used for API deployment, version management, and monitoring. These platforms offer advanced features such as caching, rate limiting, API security, and quota management. Clearly, these are must-haves if you want to be able to scale. Figure 3. Running APIs To ensure compliance with standards and guidelines established during the design phase, tools such as Stoplight's Spectral are used to perform a linting analysis of OpenAPI specifications, identifying potential problems and ensuring API consistency with design standards. And of course, at the end of the chain, you need to document your API. Tools exist to automate many tasks, such as Redocly, which generates interactive documentation from the OpenAPI Specification. The added benefit is that you ensure that your documentation is constantly up to date and always simple and readable for everyone, developers and business analysts alike. API management also involves continuous monitoring of API performance, availability, and security, as well as the timely implementation of patches and updates to ensure their smooth operation. API Analysis and Monitoring Analysis and monitoring of APIs are essential to ensure API performance, reliability, and availability. It is important to monitor API performance in real time, collect data on API usage, and detect potential problems early. The ELK Stack (Elasticsearch, Logstash, Kibana) is often used to collect, store, and analyze API access logs for monitoring performance and detecting errors. OpenTelemetry is also used in many use cases and is a must-have if you want to monitor end-to-end processes, especially ones that include an API. Regarding API performance metrics, Prometheus and Grafana are commonly used in real time, giving much information on usage trends, bottlenecks, and performance problems. FinOps and Run Management Finally, once APIs are deployed and running, it's important to optimize operating costs and monitor cloud infrastructure expenses. FinOps aims to optimize infrastructure costs by adopting practices such as resource optimization, cost forecasting, and budget management. Cloud cost monitoring tools such as AWS Cost Explorer, Google Cloud Billing, and Azure Cost Management are used to track and manage cloud infrastructure spend, keeping operating costs under control and optimizing API spend. However, in a hybrid cloud world, we could consider open-source solutions like Cloud Custodian, OpenCost, and CloudCheckr. Conclusion Obviously, you don't need to put all this in place right away to start your API journey. You have to first think about how you want to work and what your priorities are. Maybe you should prioritize design tools, like linting tools, or define your API style book and API design tool. Of course, prioritize tools that are commonly used — there's no wheel to reinvent! In fact, I'd say implement everything that is at the beginning of this toolchain because it will be easier to change after. I hope all these points in mind will enable you to get started serenely while prioritizing your own API needs. This is an excerpt from DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices.Read the Free Report
Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices. APIs play a pivotal role in the world of modern software development. Multiple types of APIs can be used to establish communication and data exchange between various systems. At the forefront lies the REST approach, which has dominated the industry due to its simplicity and scalability. However, as technology has evolved, the demands of developers and businesses have also changed. In recent years, alternatives such as GraphQL and asynchronous event-driven APIs have also emerged. They offer distinct advantages over traditional REST APIs. In this article, we will look into each of these API technologies and build a comparative understanding of them. REST: The Start of Resource-Oriented Communication REST architecture revolves around the concept of resources. These are entities that can be managed through standard HTTP methods such as GET, POST, PUT, and DELETE. One of the key characteristics of REST is its stateless nature, where each request from a client contains all the necessary information for the server to fulfill it. This decouples the client and server, allowing them to be scaled independently. Advantages and Disadvantages of REST REST APIs have some significant advantages: REST follows a simple and intuitive design based on standard HTTP methods. Each request in the REST approach is independent, resulting in better scalability and reliability. REST utilizes HTTP caching mechanisms to enhance performance and reduce the load on the origin server. REST is interoperable, working well with various programming languages and platforms due to its standard format. However, REST architecture also has several disadvantages: REST APIs can result in overfetching, where clients receive more data than needed, leading to inefficiency and waste of network bandwidth. Similar to the first point, REST APIs can also suffer from underfetching, where multiple requests are needed to fulfill complex data requirements. This results in increased latency. REST follows a synchronous approach that can lead to blocking and performance issues in high-load scenarios. Changes to the API's data schema can impact clients, resulting in tight coupling. Use Cases of REST APIs There are ideal use cases where REST APIs are much better suited when compared to other types of APIs, for example: Caching intensive applications – A read-heavy application, such as news websites or static content, can benefit from REST's caching mechanism. The standardized caching directives of REST make it easier to implement. Simple CRUD operations – When dealing with straightforward CRUD operations, REST APIs offer simplicity and predictability. Applications with a clear and static data model often find REST APIs to be more suitable. GraphQL: The Rise of Declarative Data Fetching With APIs GraphQL is a combination of an open-source language for querying data as well as a runtime for fulfilling those queries. The key principle behind GraphQL is to have a hierarchical structure for defining data queries, letting the clients precisely specify the data they need in a single request. Figure 1. GraphQL in the big picture In quite a few ways, GraphQL was a direct response to the issues with the traditional REST API architecture. However, it also promotes a strongly typed schema, offering developers a clear idea of what to expect. GraphQL supports real-time data updates through subscriptions. Over the years, a lot of work has happened on tools like GraphQL Federation to make GraphQL APIs more scalable for large enterprises with multiple domain areas. Advantages and Disadvantages of GraphQL GraphQL provides some key advantages: With GraphQL, clients can request only the specific data they need. This eliminates the overfetching and underfetching issues with REST APIs. GraphQL's strongly typed schema approach provides a clear structure and validation, speeding up development and documentation. GraphQL typically operates through a single endpoint. Clients just need to care about a single endpoint while talking to a GraphQL server even though there might be multiple sources for the data. Built-in introspection allows clients to explore the schema and discover available data and operations. There are also several disadvantages to GraphQL: Implementing GraphQL requires additional effort and expertise when compared to traditional REST APIs. Since the queries in GraphQL are flexible, caching of data can be challenging and may need custom solutions. While GraphQL reduces overfetching at the top level, nested queries can still lead to unnecessary data retrievals. Ownership of the common GraphQL layer becomes confusing, unlike the clear boundaries of a REST API. Use Cases of GraphQL There are specific scenarios where GraphQL does a better job as compared to REST APIs, for instance: Complex and nested data requirements – To fetch data with complex relationships, GraphQL helps clients precisely specify the data they need in a single query. Real-time data updates – GraphQL subscriptions help applications handle real-time data updates such as chat applications or live dashboards. With GraphQL, clients can subscribe to changes in specific data, allowing real-time updates without the need for frequent polling. Microservices architectures – In this case, data is distributed across multiple services. GraphQL provides a unified interface for clients to query data from various services. The client application doesn't have to manage multiple REST endpoints. Asynchronous APIs: A Shift to Event-Driven Architecture Over the years, the push to adopt, or migrate to, a cloud-native architecture has also given rise to event-driven architectures, the advantage being the prospect of non-blocking communication between components. With asynchronous APIs, clients don't need to wait for a response before proceeding further. They can send requests and continue their execution process. Such an approach is advantageous for scenarios that require high concurrency, scalability, and responsiveness. In event-driven systems, asynchronous APIs handle events and messages along with help from technologies like Apache Kafka and RabbitMQ, which offer a medium of communication between the message producer and the consumer. Considering a typical system using an event-driven API approach, we have producers publish events to topics, and consumers subscribe to these topics to receive and process the events asynchronously. This allows for seamless scalability and fault tolerance because both producers and consumers can evolve independently. The below diagram shows such a system: Figure 2. An event-driven system with Kafka and asynchronous APIs Advantages and Disadvantages of Asynchronous APIs There are some key advantages of asynchronous APIs: Asynchronous APIs are well suited for handling high concurrency and scalability requirements since multiple requests can be handled concurrently. Asynchronous APIs also enable real-time data processing by enabling timely response to events. Asynchronous APIs can also help better utilize system resources by offloading tasks to background processes. Lastly, asynchronous APIs increase the general fault tolerance of a system as one component failing doesn't disrupt the entire system. However, just like other API types, asynchronous APIs also have several disadvantages: There is increased complexity around message delivery, ordering, and error handling. Asynchronous APIs are more challenging to debug and test. Systems built using asynchronous APIs often result in eventual consistency, where data updates aren't immediately reflected across all components. Asynchronous APIs can also increase costs with regard to special systems for handling messages. Use Cases of Asynchronous APIs There are a few ideal use cases for asynchronous APIs when compared to REST and GraphQL APIs, including: Real-time data streaming – Asynchronous APIs are the best choice for real-time data streaming needs such as social media feeds, financial market updates, and IoT sensor data. These applications generate large volumes of data that need to be processed and delivered to clients in near real time. Integration with third-party systems – Asynchronous APIs are quite suitable for integrating with third-party systems that may have unpredictable response times or availability SLAs. Background tasks – Lastly, applications that require execution of background tasks — such as sending emails, notifications, or image/video processing — can benefit from the use of asynchronous APIs. Side-by-Side Comparison of REST, GraphQL, and Asynchronous APIs We've looked at all three types of API architectures. It is time to compare them side by side so that we can make better decisions about choosing one over the other. The table below shows this comparison across multiple parameters: Table 1. Comparing REST, GraphQL, and Async APIs Parameter REST APIs GraphQL APIs Asynchronous APIs Data fetching approach Data is fetched with predefined endpoints Clients specify the exact data requirements in the query Data is passed in the form of asynchronous messages Performance and scalability Highly suitable for scalable applications; can suffer from overfetching and underfetching problems Scalable; nested queries can be problematic Highly scalable; efficient for real-time data processing Flexibility and ease of use Limited flexibility in querying data High flexibility for querying data Limited flexibility in querying data and requires understanding of an event-driven approach Developer experience and learning curve Well established and familiar to many developers Moderate learning curve in terms of understanding the GraphQL syntax Steeper learning curve Real-time capabilities Limited real-time capabilities, relying on techniques like polling and webhooks for updates Real-time capabilities through subscriptions Designed for real-time data processing; highly suitable for streaming applications Tooling and ecosystem support Abundant tooling and ecosystem support Growing ecosystem The need for specialized tools such as messaging platforms like RabbitMQ or Kafka Conclusion In this article, we've explored the key distinctions between different API architectures: REST, GraphQL, and asynchronous APIs. We've also looked at scenarios where a particular type of API may be more suitable than others. Looking ahead, the API development landscape is poised for further transformation. Emerging technologies such as machine learning, edge computing, and IoT will drive new demands that necessitate the evolution of API approaches. Also, with the rapid growth of distributed systems, APIs will play a key role in enabling communication. As a developer, it's extremely important to understand the strengths and limitations of each API style and to select the approach that's most suitable for a given requirement. This mentality can help developers navigate the API landscape with confidence. This is an excerpt from DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices.Read the Free Report
During my early days as a Data Engineer (which dates back to 2016), I had the responsibility of scraping data from different websites. Web scraping is all about making use of tools that are automated to get vast amounts of data from the websites, usually from their HTML. I remember building around the application, digging into the HTML code, and trying to figure out the best solutions for scraping all the data. One of my main challenges was dealing with frequent changes to the websites: for example, the Amazon pages I was scraping changed every one to two weeks. One thought that occurred to me when I started reading about Large Language Models (LLMs) was, "Can I avoid all those pitfalls I faced using LLMs to structure data from webpages?" Let's see if I can. Web Scraping Tools and Techniques At the time, the main tools I was using were Requests, BeautifulSoup, and Selenium. Each service has a different purpose and is targeted at different types of web environments. Requests is a Python library that can be used to easily make HTTP requests. This library performs GET and POST operations against URLs provided in the requests. It is frequently used to fetch HTML content that can be parsed by BeautifulSoup. BeautifulSoup is a Python library for parsing HTML and XML documents, it constructs a parse tree from page source that allows you to access the various elements on the page easily. Usually, it is paired with other libraries like Requests or Selenium that provide the HTML source code. Selenium is primarily employed for websites that have a lot of JavaScript involved. Unlike BeautifulSoup, Selenium does not simply analyze HTML code: it interacts with websites by emulating user actions such as clicks and scrolling. This facilitates the data extraction from websites that create content dynamically. These tools were indispensable when I was trying to extract data from websites. However, they also posed some challenges: code, tags, and structural elements had to be regularly updated to accommodate changes in the website's layout, complicating long-term maintenance. What Are Large Language Models (LLMs)? Large Language Models (LLMs) are next-generation computer programs that can learn through reading and analyzing vast amounts of text data. At this age, they are gifted with the amazing capability to write in a human-like narrative making them efficient agents to process language and comprehend the human language. The outstanding ability shone through in that kind of situation, where the text context was really important. Integrating LLMs Into Web Scraping The web scraping process can be optimized in a great measure when implementing LLMs into it. We need to take the HTML code from a webpage and feed it into the LLM, which will pull out the objects it refers to. Therefore, this tactic helps in making maintenance easy, as the markup structure can evolve, but the content itself does not usually change. Here’s how the architecture of such an integrated system would look: Getting HTML: Use tools like Selenium or Requests to fetch the HTML content of a webpage. Selenium can handle dynamic content loaded with JavaScript, while Requests is suited for static pages. Parsing HTML: Using BeautifulSoup, we can parse out this HTML as text, thus removing the noise from the HTML (footer, header, etc.). Creating Pydantic models: Type the Pydantic model in which we are going to scrape. This makes sure that the data typed and structured conforms to the pre-defined schemas. Generating prompts for LLMs: Design a prompt that will inform the LLM what information has to be extracted. Processing by LLM: The model reads the HTML, understands it, and employs the instructions for data processing and structuring. Output of structured data: The LLM will provide the output in the form of structured objects which are defined by the Pydantic model. This workflow helps to transform HTML (unstructured data) into structured data using LLMs, solving problems such as non-standard design or dynamic modification of the web source HTML. Integration of LangChain With BeautifulSoup and Pydantic This is the static webpage selected for the example. The idea is to scrape all the activities listed there and present them in a structured way. This method will extract the raw HTML from the static webpage and clean it before the LLM processes it. Python from bs4 import BeautifulSoup import requests def extract_html_from_url(url): try: # Fetch HTML content from the URL using requests response = requests.get(url) response.raise_for_status() # Raise an exception for bad responses (4xx and 5xx) # Parse HTML content using BeautifulSoup soup = BeautifulSoup(response.content, "html.parser") excluded_tagNames = ["footer", "nav"] # Exclude elements with tag names 'footer' and 'nav' for tag_name in excluded_tagNames: for unwanted_tag in soup.find_all(tag_name): unwanted_tag.extract() # Process the soup to maintain hrefs in anchor tags for a_tag in soup.find_all("a"): href = a_tag.get("href") if href: a_tag.string = f"{a_tag.get_text()} ({href})" return ' '.join(soup.stripped_strings) # Return text content with preserved hrefs except requests.exceptions.RequestException as e: print(f"Error fetching data from {url}: {e}") return None The next step is to define the Pydantic objects that we are going to scrape from the webpage. Two objects need to be created: Activity: This is a Pydantic object that represents all the metadata related to the activity, with its attributes and data types specified. We have marked some fields as Optional in case they are not available for all activities. Providing a description, examples, and any metadata will help the LLM to have a better definition of the attribute. ActivityScraper: This is the Pydantic wrapper around the Activity. The objective of this object is to ensure that the LLM understands that it is needed to scrape several activities. Python from pydantic import BaseModel, Field from typing import Optional class Activity(BaseModel): title: str = Field(description="The title of the activity.") rating: float = Field(description="The average user rating out of 10.") reviews_count: int = Field(description="The total number of reviews received.") travelers_count: Optional[int] = Field(description="The number of travelers who have participated.") cancellation_policy: Optional[str] = Field(description="The cancellation policy for the activity.") description: str = Field(description="A detailed description of what the activity entails.") duration: str = Field(description="The duration of the activity, usually given in hours or days.") language: Optional[str] = Field(description="The primary language in which the activity is conducted.") category: str = Field(description="The category of the activity, such as 'Boat Trip', 'City Tours', etc.") price: float = Field(description="The price of the activity.") currency: str = Field(description="The currency in which the price is denominated, such as USD, EUR, GBP, etc.") class ActivityScrapper(BaseModel): Activities: list[Activity] = Field("List of all the activities listed in the text") Finally, we have the configuration of the LLM. We will use the LangChain library, which provides an excellent toolkit to get started. A key component here is the PydanticOutputParser. Essentially, this will translate our object into instructions, as illustrated in the Prompt, and also parse the output of the LLM to retrieve the corresponding list of objects. Python from langchain.prompts import PromptTemplate from langchain.output_parsers import PydanticOutputParser from langchain_openai import ChatOpenAI from dotenv import load_dotenv load_dotenv() llm = ChatOpenAI(temperature=0) output_parser = PydanticOutputParser(pydantic_object = ActivityScrapper) prompt_template = """ You are an expert making web scrapping and analyzing HTML raw code. If there is no explicit information don't make any assumption. Extract all objects that matched the instructions from the following html {html_text} Provide them in a list, also if there is a next page link remember to add it to the object. Please, follow carefulling the following instructions {format_instructions} """ prompt = PromptTemplate( template=prompt_template, input_variables=["html_text"], partial_variables={"format_instructions": output_parser.get_format_instructions} ) chain = prompt | llm | output_parser The final step is to invoke the chain and retrieve the results. Python url = "https://www.civitatis.com/es/budapest/" html_text_parsed = extract_html_from_url(url) activites = chain.invoke(input={ "html_text": html_text_parsed }) activites.Activities Here is what the data looks like. It takes 46 seconds to scrape the entire webpage. Python [Activity(title='Paseo en barco al anochecer', rating=8.4, reviews_count=9439, travelers_count=118389, cancellation_policy='Cancelación gratuita', description='En este crucero disfrutaréis de las mejores vistas de Budapest cuando se viste de gala, al anochecer. El barco es panorámico y tiene partes descubiertas.', duration='1 hora', language='Español', category='Paseos en barco', price=21.0, currency='€'), Activity(title='Visita guiada por el Parlamento de Budapest', rating=8.8, reviews_count=2647, travelers_count=34872, cancellation_policy='Cancelación gratuita', description='El Parlamento de Budapest es uno de los edificios más bonitos de la capital húngara. Comprobadlo vosotros mismos en este tour en español que incluye la entrada.', duration='2 horas', language='Español', category='Visitas guiadas y free tours', price=27.0, currency='€') ... ] Demo and Full Repository I have created a quick demo using Streamlit available here. In the first part, you are introduced to the model. You can add as many rows as you need and specify the name, type, and description of each attribute. This will automatically generate a Pydantic model to be used in the web scraping component. The next part allows you to enter a URL and scrape all the data by clicking the button on the webpage. A download button will appear when the scraping has finished, allowing you to download the data in JSON format. Feel free to play with it! Conclusion LLM provides new possibilities for efficiently extracting data from non-structured data such as websites, PDFs, etc. The automatization of web scraping by LLM not only will save time but also ensure the quality of the data retrieved. However, sending raw HTML to the LLM could increase the token cost and make it inefficient. Since HTML often includes various tags, attributes, and content, the cost can quickly rise. Therefore, it is crucial to preprocess and clean the HTML, removing all the unnecessary metadata and non-used information. This approach will help use LLM as a data extractor for webs while maintaining a decent cost. The right tool for the right job!
Marty Cagan describes the job of the Product Manager as “to discover a product that is valuable, usable, and feasible." Finding the balance between the business, users, and technology demands a diverse skill set. There are many things that are going on simultaneously that require attention. In this regard, Jira is great. Sure, it has some downsides, but the tool can help Product Managers to: Keep the product strategy aligned. Clearly prioritize tasks while keeping them structured and organized. Analyze the performance of your team. Use a Roadmap To Keep Your Strategy Aligned As powerful as Jira is in the right hands, it is not a solution for everything. For instance, it is probably not the best tool for developing a product roadmap. However, it is quite good for managing one. What this means is that Jira has great functionality for managing roadmaps in a quick, actionable, and transparent way. Nevertheless, it requires proper input: You need to break down your scope into Epics and tasks before you start building a roadmap in Jira. We typically use a framework called BRIDGeS for multi-context analysis of a project. This framework leaves us with prioritized, ready-to-use Epics and tasks at the end of the session. Given this article is not about roadmaps per se, I would rather not go into too much detail. I will be focusing on Jira instead. Setting Up a Timeline in Jira Once you have your work broken down into Epics and tasks, creating a roadmap – or, as Jira calls it, a Timeline – is quite simple. Navigate to your board. Select the “Timeline” option from the menu on the right. Click on the “+ Create Epic” to add an Epic. Add child issues by clicking on the “+” sign next to the Epic. Click on the timeline to set the timeframe for the Epic. Tips and Tricks for Using Jira’s Timeline Feature Unlike most Jira features, the Timeline is rather intuitive and friendly to new users. Still, there are certain easily missable tips and tricks that can make your job much simpler. It’s just that you need to know where to look Add Dependencies You can add Dependencies between Epics from the Timeline. Simply hover over the timeline for an Epic and you will see two dots – one at the top right corner and one in the bottom left corner. Click and drag them to link one Epic to another. This is useful for understanding the order of work or visualizing potential blockers. Note: The color of the connective thread will change to red if the dates of Epics overlap. This feature is quite handy for easily seeing if certain dependencies are becoming blockers. Still, I’d recommend using dependencies wisely, otherwise the roadmap will become confusing because of the intertwined Epics. Use Different Colors for Epics You can right-click on the timeframe to easily change the color of an Epic or to remove start and end dates. Color-coding your Epics is a useful element of visualization. View Settings You can adjust the settings of the timeline if you wish to filter out certain completed issues or expand/collapse all of the Epics at the same time. Another useful option you can find in the view settings is the progress bar. Enable it to see a bar indicating the progress of an Epic. Filter out Epics With a Certain Status You can use the status category filter to hide the Epics and tasks that are marked as done from the timeline. This simple filter greatly improves the visibility of the roadmaps for times when you need to review done/in progress/future scope. Prioritize and Manage Tasks in the Backlog Now that we have an actionable plan, let’s take a look at how Jira can be used to execute it. Setting Up a Backlog in a Kanban Project In my experience, most Agile teams prefer to use a Scrum board that has the backlog feature enabled by default. That being said, a Kanban board needs a little bit of tweaking if you want to have a separate backlog rather than storing all of your issues on the board. The task of adding a backlog is slightly simpler for Team-Managed projects. Simply select the Add View option from the left side panel and enable the backlog. The process of adding the backlog in a Company-Managed project is a bit trickier. Go to the three dots menu at the top right corner of your board. Select Board settings. Select the Columns option. Drag the backlog status card from the board and into the Kanban backlog column. Delete the original Backlog column by clicking on the trash bin icon. Going back to the board, you’ll see that it has only three columns left, and the backlog has been moved to the side panel. Hint: This approach has an added benefit. Creating issues from the Backlog screen is much simpler and faster than from the board. Just click on the + Create Issue button and type in the name of your task. You can keep on typing and hitting enter to add new issues. And you can change their type as well. Setting Up a Backlog (Or Several) in a Scrum Project As I mentioned earlier, the Scrum project comes with the backlog feature enabled by default. That said, there is a major difference between the backlogs in Scrum and Kanban Jira projects: a Scrum Project has two backlogs by default. One is the Project Backlog and the other is the Sprint Backlog. The Sprint Backlog consists of a set of user stories or tasks that the development team commits to completing within a specific sprint or time-boxed iteration. It is a subset of the product backlog and represents the work sprint planning for that particular sprint. The Product Backlog contains a prioritized list of all the desired features, enhancements, and bug fixes for the product. It represents the complete scope of work that needs to be done over multiple sprints. Hint: The backlog view in Jira allows you to create several Sprints. These Sprints can be used as separate backlogs for certain specific tasks. For example, you can use these Sprints as separate backlogs for Bugs, Support Requests, the Icebox, etc. This functionality is super handy for keeping your work well-organized. Plus, this approach allows you to keep your work well-organized. The tasks from these backlogs can be pulled into the Sprint Backlog during the Sprint Planning Session. Story Points As a feature, Story Points are used to estimate the complexity of a user story. Typically, we use the following approach when it comes to assigning points to user stories: Point Description 1 One-liner change. You know what should be changed. Very easy to test. 2 You are aware of what to do. Changes are bigger than one-liner~1-2 days to implement. May include regression testing 3 Bigger scope. May require some research/documentation reading/codebase research. Includes unknown parts. 5 Biggest story. Not enough to split. 8 Must be split. Do research first. Bonus Tip: Backlog Refinement Backlog refinement is the process of reviewing, prioritizing, and tidying up the backlog. It is a necessary activity as, over time, people will add a lot of tasks that are missing context. For now, let’s focus on the benefits of tidying up your tasks: The team is working on the tasks that are adding real value to the product. The tasks are optimized and broken down in a way that a single issue doesn’t take longer than an entire Sprint. The work that is in progress reflects the roadmap. How do we do it? We typically refine the backlog once every two weeks. We take the stories from the product backlog and place them into relevant Sprint containers like Bugs, Technical Debt, or upcoming Sprint. We review the estimation and priority of the tasks that are being moved from the Product Backlog. Analyze the Performance of Your Team With the Built-In Reports Jira has a variety of reporting tools that are available to Product Managers. They are easily accessible from the reports tab on the right-side menu. Note: The Reports tab may not be enabled for you by default. Therefore, please follow these steps in case you do not see it: Select the Add View option. Select the More Features option. Find the Reports option and toggle it on. These reports can be used to analyze the performance of your team. They are also easily shareable and exportable. There is a wide selection of reports, but using all of them isn’t necessary. Here is a brief overview of several reports that we find to be the most useful: Burndown chart: Tracks the remaining story points in Jira and predicts the likelihood of completing the Sprint goal. Burnup chart: Tracks project progress over time and compares the work that is planned to the work that has been completed to date. Sprint report: Analyzes the work done during a Sprint. It is used to point out either overcommitment or scope creep in a Jira project. Velocity chart: This is a kind of bird’s eye view report that shows historical data of work completed from Sprint to Sprint. This chart is a nice tool for predicting how much work your team can reliably deliver based on previously burned Jira story points. Conclusion There are many new, slick work management tools on the market. Most are probably better than Jira in terms of UI and UX. That being said, as one of the oldest solutions out there, Jira has had the time and resources to develop a wide selection of features. This is why many PMs feel lost and confused when they are experiencing Jira for the first time. Don’t worry though – we’ve all been there. That’s why this little guide exists, showing you the different options of tools that will work best for you. Consider this to be your starting point in the endless sea of Jira features.
System-Level Scrum Stakeholder Anti-Patterns
May 1, 2024 by CORE
Reliability Models and Metrics for Test Engineering
May 1, 2024 by CORE
Exploring FAPI 2.0: The Latest in API Security Technology
May 1, 2024 by
Explainable AI: Making the Black Box Transparent
May 16, 2023 by CORE
Scrolling With Konva.js and React
April 30, 2024 by
The Evolution of Platform Extensibility
April 30, 2024 by
Five IntelliJ Idea Plugins That Will Change the Way You Code
May 15, 2023 by