Patrick Mouwen — PMN Consulting

D365 Commerce CSU V47/9.57 Unofficial Release Notes

Thu, 02 Apr 2026 00:00:00 GMT

If you work with D365 Commerce — as an architect, developer, ISV, or integration specialist — you know the feeling. A new version drops, the official docs give you the feature headlines, and then you're left figuring out the rest: what actually changed in the code and APIs? Which schemas gained new fields? What are those new flight keys doing? Where is the product really heading?

That's exactly why I started publishing these unofficial Commerce CSU release notes.

Every release, I pull apart the Commerce Scale Unit: decompile the assemblies, diff the OpenAPI specs, trace the new code paths, and map it all to what Microsoft shared about the release. The goal is simple — give the community the technical depth that doesn't exist anywhere else.

I've been building on Commerce CSU since around 2015. Over 650 APIs, B2C and B2B, headless and POS — this platform is deep, and it deserves documentation that matches.

This edition — V9.57 / 10.0.47 — is the most architecturally significant release I've analyzed. Here's what's inside.

𝘛𝘩𝘪𝘴 𝘢𝘳𝘵𝘪𝘤𝘭𝘦 𝘳𝘦𝘧𝘭𝘦𝘤𝘵𝘴 𝘮𝘺 𝘪𝘯𝘥𝘦𝘱𝘦𝘯𝘥𝘦𝘯𝘵 𝘵𝘦𝘤𝘩𝘯𝘪𝘤𝘢𝘭 𝘢𝘯𝘢𝘭𝘺𝘴𝘪𝘴 𝘰𝘧 𝘴𝘰𝘧𝘵𝘸𝘢𝘳𝘦 𝘢𝘳𝘵𝘪𝘧𝘢𝘤𝘵𝘴. 𝘐𝘵 𝘪𝘴 𝘯𝘰𝘵 𝘦𝘯𝘥𝘰𝘳𝘴𝘦𝘥 𝘣𝘺 𝘰𝘳 𝘢𝘧𝘧𝘪𝘭𝘪𝘢𝘵𝘦𝘥 𝘸𝘪𝘵𝘩 𝘔𝘪𝘤𝘳𝘰𝘴𝘰𝘧𝘵. 𝘍𝘶𝘭𝘭 𝘥𝘪𝘴𝘤𝘭𝘢𝘪𝘮𝘦𝘳 𝘢𝘵 𝘵𝘩𝘦 𝘦𝘯𝘥 𝘰𝘧 𝘵𝘩𝘪𝘴 𝘢𝘳𝘵𝘪𝘤𝘭𝘦.

A-Overview of this Release

The 9.57 / 10.0.47 Release is defined by 3 architectural shifts: the arrival of MCP (Model Context Protocol) as a first-class CSU endpoint, a fundamental identity model change for B2B Commerce, and the continued maturation of the headless payment and POS modernization story.

The headline is the MCP server embedded directly into the CSU. Microsoft has shipped an MCP endpoint (still in private preview!) with API key authentication, structured tool definitions, and logging. It exposes Commerce capabilities (product search, cart management, payment link creation) to AI agents and agentic platforms. This is not a bolt-on: the CSU now ships with ModelContextProtocol assemblies, AI Abstractions, and Server-Sent Events transport. If you are building agentic commerce experiences — conversational shopping, AI-assisted ordering, or any scenario where an LLM needs to interact with Commerce business logic — the infrastructure is now in the box and ready to be enabled.

The second major shift is the B2B multi-outlet model. Commerce is moving from Customer (Person) type to Contact type for B2B buyers. This is a deep, pervasive change: ContactPersonId now flows through Cart, SalesOrder, security checks, and the entire cart-to-order lifecycle. New APIs support contact retrieval, identity linking, and organization-level catalog access. A feature flag (RetailB2BContactMultiOutletOrderingFeature) gates the rollout, and migration scripts are being provided — but this is a one-way street once enabled. If you are in B2B Commerce, this is the most impactful change to your data model in years.

Third, the POS modernization journey reaches the product listing page, product detail page, and transaction page — all now migrated to React and Fluent UI. What makes this release different from a typical UI refresh is the API surface behind it: eight new VisualProfile flags give you register-level control over which insights (inventory, discounts, product suggestions) are surfaced, and a new promotion evaluation endpoint (GetAvailablePromotionsByTransactionJson) lets the React front-end evaluate discount eligibility without a full server-side cart round-trip.

Alongside these three themes, the Unified Pricing engine (GUP) receives its heaviest investment yet (10 new flight keys), the cross-legal entity inventory story lands with a dedicated API for multi-company store search, and the telemetry stack gets a complete overhaul — moving from the older infrastructure to Azure Data Explorer / Kusto-based observability.

B-Resources

V9.57 Commerce CSU API explorer (React control with filter/search): </tools/csu-api-explorer-v57.html>
V9.57 full Swagger/OpenAPI spec (direct download): </tools/9.57.26071.1.json>

C — API and Schema Changes

C.1 New APIs

C.2 Removed APIs

None.

C.3 Data Model Changes

C.3.1 B2B Multi-Outlet/Contact model

Cart and SalesOrder gained ContactPersonId — this is the primary identifier linking a transaction to the specific B2B contact who placed it, distinct from the organization customer account.
BusinessPartner gained OrganizationCustomerAccountNumber, Addresses[], and SpendingLimit — enriching the organization-level entity with account resolution and spending governance data.
BusinessPartnerUser gained ContactPersonId, IsEnabled, OrganizationCustomerAccountNumber, and Phone — the user entity now bridges to the Contact model with explicit activation state and org-level tracing.
New entity: BusinessPartnerContact (PartyRecordId, IsActivated, FirstName, LastName) — a lightweight Contact representation used by the GetContact API.
New entity: LinkToExistingDirPartyResult (PartyRecordId, ExternalIdentityLinkRecordId, FirstName, LastName, EmailAddress) — the result of identity-to-party linking during migration.
New security claim: ContactPersonIdClaimType added to the claims system — the Contact identity is now a first-class authentication claim alongside CustomerId and StaffId.
New role: "BusinessPartnerContact" added as a recognized principal role.

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: This is the API and data model surface for the fundamental B2B identity shift from Customer (Person) to Contact. A single contact can buy for multiple organizations, orders are created against the organization account (not the individual), and pricing/credit limits/catalogs are driven by the organization. The migration path (CreateLinkToExistingParty) ensures existing B2B users retain their external identity mappings when transitioning.

C.3.2 Cross-Legal Entity Inventory

OrgUnitLocation gained DataAreaId — stores and warehouses now carry their legal entity identifier in the API surface.
OrgUnitAvailabilitySearchCriteria gained DataAreaIds[] — the search criteria can now specify which legal entities to include.
New feature flag: CrossCompanyInventoryLookupFeatureName gates the capability.

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: Enables POS users to see inventory across legal entities in the same fulfillment group. Backed by Inventory Visibility Service (IVS). This may be extended to e-commerce, DOM cross-LE fulfillment with intercompany orders, and decoupled ICO lifecycle in the future.

C.3.3 Asynchronous payments/Pay-by-link

AsynchronousPaymentParameters gained ReturnUrl and ShopperLocale — supporting proper redirect-based payment flows and locale-aware payment page rendering.
CreatePaymentLinkParameters gained the same ReturnUrl and ShopperLocale fields.

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: Continuation of the V46 async payment maturation. ReturnUrl enables proper redirect-back after payment completion (critical for QR code / pay-by-link flows where the customer completes payment on a different device or browser). ShopperLocale ensures the payment page renders in the customer's language — directly supporting the Adyen connector QR code feature announced for 10.0.47

C.3.4 POS Modernization - VisualProfile feature flags

VisualProfile gained eight new nullable boolean flags:

ShowDimensionPickerEnabled
ShowInlineQtyDiscountMsgEnabled
ShowNearbyInventoryEnabled
ShowProductSuggestionsEnabled
ShowQuantityDiscountCaptionsEnabled
ShowQuantityDiscountsEnabled
ShowStockCountAndAvailabilityEnabled
ShowThresholdDiscountBarEnabled

Where it shows up: All endpoints that return VisualProfile (device configuration, register setup).

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: These are the register-level feature flags for the React/Fluent UI POS modernization. Each flag controls a specific insight or UI element on the modernized search results, PDP, or transaction pages. This gives retailers fine-grained control over rollout — you can enable inventory insights for one register while keeping it off for others, allowing a phased transition and A/B testing of the new experience.

C.3.5 Self-Checkout Language Selection

DeviceConfiguration gained EnableLanguageSelectionOnSelfCheckoutWelcomePage (boolean), SecondaryLanguages (string[]), and OfflineSQLiteDbExport (boolean).

Where it shows up: Device configuration endpoints.

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: EnableLanguageSelectionOnSelfCheckoutWelcomePage and SecondaryLanguages power the multilingual self-checkout experience. The customer can switch language on the welcome screen, and all UI elements update instantly.

C.3.6 Address Purposes

Address entity gained AddressPurposes (IList of AddressPurpose) — enriching addresses with their associated purpose values (Business, Delivery, Home, etc.).

Where it shows up: This is the single most widespread change in the diff by surface area — virtually every endpoint that returns an Address object now includes AddressPurposes. The change is controlled by the UseLogisticsLocationMultiRoleTypesFlight.

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: Addresses can serve multiple purposes simultaneously. Previously, purpose was inferred or required separate lookups. Now it's inline — improving address selection logic for delivery, invoicing, and B2B scenarios where a single address may serve both business and delivery purposes.

C.3.7 Unified Pricing - PricingComponentCodeConcurrencyMode

New enum PricingComponentCodeConcurrencyMode with values: None, Compounded, BestPrice, NeverCompound.

Where it shows up: In the pricing engine's price adjustment evaluation pipeline, controlling how pricing components interact across priorities.

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: Extends the GUP concurrency model with a NeverCompound mode — allowing pricing rules to explicitly prevent compounding with other adjustments. This is part of the broader Unified Pricing maturation that includes fixed price adjustment support, tender discounts, multi-currency conversion, and enhanced concurrency modeling.

C.3.8 MCP Infrastructure

New schema McpException (extends CommerceException) — the first appearance of MCP error handling in the CSU's public schema.

Where it shows up: Not exposed as an endpoint in the OData Swagger — the MCP server runs on a separate /mcp endpoint using Server-Sent Events (SSE) transport.

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: While the MCP endpoint itself is not part of the OData API surface, the exception type landing in the public schema signals that MCP is now a supported, (soon) production-grade component of the CSU. See Section D for the full MCP code analysis.

C.3.9 Telemetry

EnvironmentConfiguration gained ClientKustoTelemetryKey, HardwareStationKustoTelemetryKey, and KustoTelemetryEndpoint.

Where it shows up: Environment configuration endpoints.

𝐋𝐢𝐤𝐞𝐥𝐲 𝐢𝐧𝐭𝐞𝐧𝐭: The CSU is moving to Azure Data Explorer (Kusto) based telemetry. A new Microsoft.Dynamics.Retail.Diagnostics.Kusto assembly ships with V47, alongside Microsoft Unified Telemetry libraries. This replaces the older telemetry infrastructure with direct Kusto ingestion — enabling richer, queryable observability for Commerce operations.

D —Code Changes

This section goes beyond the API surface and examines changes in the decompiled CSU assemblies (the actual Commerce Scale Unit runtime code). 287 assemblies in V47 vs 265 in V46.

D.1 MCP Server - Full implementation in the box

The most significant code-level finding. The CSU now ships with a complete MCP (Model Context Protocol) server implementation embedded directly in Microsoft.Dynamics.Retail.RetailServerLibrary.AspNetCore.

New namespaces:

Microsoft.Dynamics.Retail.RetailServerLibrary.Mcp
.Mcp.Configuration
.Mcp.Contracts
.Mcp.Converter / .Converters
.Mcp.Exceptions
.Mcp.Middlewares
.Mcp.Tools

Three tool classes ship out of the box (may be extended):

Configuration via app settings:

𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲: Full API key authentication middleware (McpApiKeyAuthenticationMiddleware) with named client support. Comprehensive structured logging via McpEvents enum covering authentication, tool call lifecycle, and errors.

𝐄𝐫𝐫𝐨𝐫 𝐡𝐚𝐧𝐝𝐥𝐢𝐧𝐠: Custom McpException with domain-specific error codes: InvalidChannelId, InvalidCartId, InvalidProductId, InvalidQuantity, CartLineNotFound, CustomerNameRequired, EmailRequired, StreetRequired, CityRequired, StateRequired, ZipCodeRequired, CountryOrRegionCodeRequired, CartHasNoLines, CartMissingShippingAddress, DeliveryModeRequired, DeliveryModeDescriptionNotFound, CartOperationRetryFailed.

𝐍𝐞𝐰 𝐚𝐬𝐬𝐞𝐦𝐛𝐥𝐢𝐞𝐬 𝐬𝐮𝐩𝐩𝐨𝐫𝐭𝐢𝐧𝐠 𝐌𝐂𝐏: ModelContextProtocol, ModelContextProtocol.Core, ModelContextProtocol.AspNetCore, Microsoft.Extensions.AI.Abstractions , System.Net.ServerSentEvents.

D.2 B2B Contact Model - Deep Pervasive Implementation

10+ new data request types span the full data access layer:

GetBusinessPartnerContactByContactPartyDataRequest
GetBusinessPartnerContactByContactPersonIdDataRequest
GetBusinessPartnerContactByEmailDataRequest
GetBusinessPartnerContactByExternalIdentityDataRequest
GetBusinessPartnersByContactPartyDataRequest
GetBusinessPartnerUserByContactPersonIdDataRequest
GetBusinessPartnerUserByOrganizationCustomerAndContactPartyDataRequest
GetBusinessPartnerCatalogsByContactPersonIdDataRequest
GetBusinessPartnerByOrganizationCustomerAccountNumberDataRequest
CreateLinkToExistingPartyDataRequest

The feature flag 𝐑𝐞𝐭𝐚𝐢𝐥𝐁2𝐁𝐂𝐨𝐧𝐭𝐚𝐜𝐭𝐌𝐮𝐥𝐭𝐢𝐎𝐮𝐭𝐥𝐞𝐭𝐎𝐫𝐝𝐞𝐫𝐢𝐧𝐠𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐍𝐚𝐦𝐞 is checked throughout the codebase — customer service, security, workflow, and catalog resolution all branch between the old Person path and the new Contact path. Cart access security now validates ContactPersonId as a third dimension alongside CustomerId and StaffId.

D.3 Bing Search Removed

Microsoft.Dynamics.Commerce.Runtime.SearchServices.Bing assembly has been removed from V47. Product search is now exclusively handled by Azure Search / Azure AI Search.

D.4 Telemetry Stack Overhaul

New assembly: Microsoft.Dynamics.Retail.Diagnostics.Kusto — a Kusto logger implementation for Retail components. Accompanied by 6 Microsoft Unified Telemetry assemblies (Microsoft.Unified.Telemetry, .OpenTelemetrySink, .ContractsAndContextProviders, Microsoft.Office.Telemetry.EventFlags, Microsoft.Office.PrivacyGuard). This is a platform-level shift to structured, Kusto-queryable telemetry.

D.5 Unified Pricing Engine - Significant Evolution

Key code-level changes in the pricing engine:

New PricingComponentCodeConcurrencyMode with NeverCompound support
GetAttributeTenderDiscounts method added — GUP now supports tender-type (payment-method) discounts via ReadGlobalUnifiedPricingTenderDiscounts
New GetBasePriceTypeDataRequest/Response — base price type resolution by price tree name
Generic currency support with exchange rate type conversion — multi-currency pricing across legal entities
Fixed price adjustment support (OfferPrice method) gated by GUPSupportFixedPriceAdjustmentFlight
Unit of measure conversion caching for price adjustments
Enhanced discount header amount currency conversion
V2 sproc for non-coupon discount category lookups (performance)

E-New Flight Keys

Microsoft implements new CSU flight keys with every new CSU version. These undocumented flight keys steer specific CSU behavior. V9.57/10.0.47 adds 29 new flight keys (picking up from the last V46 key at index 151).

F - Disclaimer

The content of this article represents my independent technical analysis and personal interpretation of publicly available software artefacts. It is not endorsed by, affiliated with, or authorised by Microsoft Corporation. Nothing in this article constitutes official Microsoft documentation, guidance, or product commitments.

All findings are derived from analysis of the software and public documentation. Feature descriptions, presumable use cases, and "likely intent" assessments reflect my professional opinion based on observable code and API changes — they may not reflect Microsoft's actual design intent, planned behaviour, or final implementation.

Information regarding features described as preview, private preview, or roadmap items is subject to change at Microsoft's sole discretion and should not be relied upon for planning, purchasing, or architectural decisions.

This article is provided for informational and educational purposes only, on an "as-is" basis without warranties of any kind, express or implied. The author assumes no liability for any direct, indirect, incidental, or consequential damages arising from the use of or reliance on the information contained herein, including but not limited to damages resulting from technical decisions, implementation choices, or system behaviour based on this content.

For authoritative product information, consult Microsoft's official documentation at learn.microsoft.com.

The middle layer is 50% of your transformation

Thu, 26 Mar 2026 00:00:00 GMT

What a detour into food certification taught me about ERP, monoliths, and agentic AI

I spend most of my time in Retail, Wholesale, Manufacturing, and Supply Chain. Dynamics 365, Commerce platforms, Enterprise architectures, Integration architectures — that’s my world. So when I recently took on a project in professional services — specifically in the Testing, Inspection & Certification (TIC) industry — I expected a completely different landscape.

What I found instead was remarkably familiar.

Within a day of research, I recognised the same architectural patterns, the same monolithic constraints, and the same integration challenges I’ve been solving in Retail and Manufacturing for years. And the solutions? Almost identical.

That’s worth writing about, because I think there’s a broader lesson here for anyone struggling with the question of how to make their Enterprise Architecture Agentic AI ready.

The monolith that everyone trusts

In the TIC world, certification bodies rely on a specialised ERP system to manage the full lifecycle of audits, inspections, and certifications. Think of it as the equivalent of Dynamics F&O or SAP, but built specifically for testing and certification workflows.

This system is deeply trusted. It’s the single version of the truth. It’s 100% auditable. The organisation’s accreditation depends on it. People have built their careers around it. The former CEO introduced it. It has decades of credibility.

Sound familiar? That’s exactly how enterprises describe their ERP. And for good reason — these systems are reliable, comprehensive, and proven. They earn their monolithic status through years of delivering on their core promise: be the system of record.

The problem: monoliths don’t speak event

Here’s where it gets interesting. The TIC industry is being disrupted by scheme owners — the organisations that define certification standards — launching their own modern cloud platforms with comprehensive REST APIs. These new platforms support headless execution, event-driven workflows, and full lifecycle management via API.

But the trusted monolith at the heart of the certification body? It exposes a basic CRUD API layer. No webhooks. No event notifications. No publish/subscribe. No way to say “tell me when a quote gets approved” and have the system push that notification/

And that’s a problem, because the future everyone is talking about — Agentic AI — lives and breathes on deterministic, event-driven APIs. An AI agent that needs to “check if the quote was approved, then trigger the planning workflow, then notify the auditor” cannot poll a CRUD endpoint every five minutes and hope for the best. It needs events. It needs a contract. It needs MCP.

The middle layer: where transformation actually happens

Here’s the insight I keep seeing confirmed, now across two completely different industries: the transformation isn’t in the monolith. It’s in the layer between the monoliths.

Many implementation partners still think that ERP (or TIC platform, or CRM) is 70–80% of a transformation program. My experience says it’s shifting. The middle layer — the operational data platform with workflow orchestration and AI enablement — is becoming 50% of the programme. And growing.

What does this middle layer look like in practice?

1. Data Layer: an operational database — not a data warehouse

Extract the relevant entities and status changes from the monolith on a regular schedule. Not a full canonical model for BI. Not a data lake. A focused, pragmatic operational database that stores only what’s needed for cross-system integration: business partners, work orders, approvals, key identifiers, status timestamps.

The key insight: if your monolith can’t emit events, you construct the events yourself by detecting state changes in the extracted data. A quotation that moved from “draft” to “accepted” between two extraction cycles? That’s an event. A work order that changed status from “in review” to “approved”? That’s a trigger. You don’t need the monolith to tell you — you can see it in the data.

2. Workflow Layer: a workflow orchestration layer

Logic Apps, Power Automate, Azure Functions, Service Bus — whatever your stack, the orchestration layer sits on top of the operational database and reacts to the constructed events. When it detects that a quote was approved, it calls the external platform’s API to register the client. When a certification decision is made, it pushes the outcome to the scheme owner’s registry.

This is where business process logic lives — not in the monolith (which is too rigid) and not in the external platform (which you don’t control). The orchestration layer is yours. You own it, you design it, you evolve it.

3. Action Layer: An MCP surface for agentic AI

If your monolith’s CRUD APIs are exposed through Azure API management, they can be wrapped as MCP-enabled tool endpoints. Combined with CRM, F&O, and external platform endpoints on the same gateway, you create a unified tool surface for AI agents.

Suddenly, a Copilot agent can query a client’s certification status from the TIC system, check their outstanding invoices in F&O, look up their contact history in CRM, and draft a follow-up — all through one protocol. The monolith doesn’t need to be rebuilt. It just needs to be accessible.

4. Intelligence layer: the brains for the 'feedback loop'

Here’s where it comes together. Relevant operational data flows in, is combined with BI insights and AI powered predictive analytics, actions are distilled, and with MCP those actions are executed across systems. Collect, analyse, act — all through the middle layer.

Over time, fewer and fewer employees will work in the monoliths - They operate in the middle layer instead, relying on Agentic AI being the 'strings' to direct the System of record 'puppets'.

The shift: let monoliths be monoliths

I know that sounds circular, but hear me out. For decades, we’ve expected our ERP to be everything: system of record, workflow engine, reporting tool, collaboration platform, and now AI host. That’s too much weight for any monolith.

The shift I see — confirmed in retail, manufacturing, and now in TIC — is that monoliths are settling into what they’re actually good at: being the trusted, auditable, comprehensive system of record. They keep their robustness. They keep their reputation. They keep their regulatory compliance.

But the workflow-oriented tooling, the approval flows, the cross-system orchestration, the AI-enabled intelligence — that doesn’t need to be built inside the monolith. It lives in the middle layer. And increasingly, it’s the middle layer that users actually interact with day to day.

The result? Fewer users logging into ERP. Fewer users logging into the TIC platform. These systems become the trusted archive, the system of record. The middle layer becomes the operational face — where work gets done, decisions get made, and AI agents collaborate with humans.

Agentic AI acts as the strings to direct the puppets, the systems of record which now become systems of action.

Best of all worlds

This isn’t a replacement strategy. It’s an augmentation strategy. The monolith retains its strengths. The middle layer compensates for its limitations. And the organisation gets an architecture that’s ready for agentic AI without ripping out the systems everyone trusts.

I found it striking that a step into a completely different industry confirmed what I’ve been seeing in retail and commerce for years. The same monolithic patterns. The same CRUD API constraints. The same event-driven gaps. And the same solution: build the middle layer, make it intelligent, and let the monoliths do what they do best.

If your organisation is asking “how do we make our architecture agentic AI ready?” — the answer probably isn’t “replace the ERP.” or to put your software vendor under pressure. It’s build the layer between your systems. That’s where 50% of your transformation lives. And that’s where the future is being built.

—

Designing Headless Commerce Starts with Data — Not Apps

Wed, 04 Feb 2026 00:00:00 GMT

It doesn’t matter if I speak to e-commerce gurus or leaders in App development, they all say the same: the difficulty is not making the Apps or customer journeys. The difficulty is in getting the right data, in the right quality.

This is even more urgent in Headless setups. Headless Apps do not have any data or local business logic. They live by data and services from backend systems like Dynamics 365 ERP and Dynamics 365 CE (CRM).

In this blog post, we’ll see how we can overcome these hurdles.

Hurdle 1 – Exposing the right data right

In the ideal world, UX designers are in the lead when Apps, portals or any customer oriented systems are built. Side note here is that I recommend to converge customer and employee oriented apps, but let’s put that aside. In how many Dynamics 365 implementation projects are UX designers in the lead when it comes to customer oriented scenarios? I didn’t come across many in my 20-year career.

Let’s revert the question. How can we put UX designers in the lead?

My answer:

1. Start with the journey. Let the UX designers design the optimal customer journey first and then map to Data and Services. Not the other way around. Literally map a wireframe to APIs. Please note that this requires your API providers to be very experienced in both the backend system (like D365 ERP) and API domain.

2. Enterprise API library. Publish core/key APIs in an enterprise API library. Ideally even before wireframe design begins. My best practice as I already described in my blog post in 2022: describe your D365 APIs in the OpenAPI standard and publish them on Azure API Management.

3. Publish the OpenAPI specifications. Use APIM Developer portal or Swagger portal to deliver the API specs to your App/portal developers. See some examples below which combine APIs based on D365 Commerce and D365 ERP:

4. Composite APIs. In many cases, UX designers design grids with data which is exposed by multiple APIs. For example, Sales order line information combined with product data. My best practice: combine them in a composite API to make life easier for the developers. The “individual” APIs and “composites” can co-exist in your API library!

With the wireframe mappings and documented APIs in the bag, frontend developers can now automatically generate code to consume the APIs. This will allow them to autonomously support your UX designers!

Hurdle 2 – Exposing data in the right quality

If your PDM/PIM data does not contain the right product descriptions, categorizations or translations, it can negatively impact your customer experience. No matter how strong your UX/customer journey is designed and implemented. In headless Apps, frontend journeys may even get broken returning no data at all!

So, the key question is: how can we guarantee data quality? My best practices:

1. Identify your critical data. Identify backend setup and master data which is critical for your customer journeys. You can easily derive them from your wireframe/API mappings. Map these requirements back to the data and processes in D365 ERP and CRM or other backend Apps.

2. Prevent errors by a strong process foundation. Educate the responsible business owners properly. Make them understand what the impact of their actions is on data quality.

3. Deploy a Data Quality framework. Utilize Synapse link or Fabric link to export your data to Azure Data Lake. Use Azure Data Factory to process the data and identify errors and warnings. Write these errors and warnings into tables. Surface the errors and warnings in Power BI Reports. This is your feedback loop into #2, the BAU processes in your organization.

4. Deploy a Data Quality agent. Let Data Quality agents work with the errors and warnings to assist your users to resolve the error. Agents could suggest data actions, such as unpublishing items from assortments if they are missing critical data or prices. As a next step, Agents could even resolve the actual issues utilizing Dynamics 365 MCP server.

About the Author

The further I progress in my career, the more I see that the most successful IT projects aren’t feature driven. The most successful projects have a holistic view, acknowledging that downstream success can depend on upstream quality and structure.

Connect with me directly on LinkedIn if you want to know more.

A language trick for headless commerce

Fri, 23 Jan 2026 00:00:00 GMT

If you follow my quarterly unofficial release notes for D365 Commerce CSU (example link), you've probably found a link to my website which hosts the latest OpenAPI documentation for the 650 Commerce CSU APIs:

In a truly headless environment, you'd want some of these APIs to adapt content based on user language. For example, when a customer user clicks a language flag in a portal to change from English to German, you'd want the content to return product names or mode of delivery descriptions in German, right?

If you look into the API request definitions, you'll find language related properties in some of the APIs. But what the Microsoft documentation doesn't tell you is that passing an Accept-Language header actually steers the response content - That's our hidden gem for today.

What CSU does with Accept-Language

When you add the header, CSU will return certain translated values in that language (when translations exist and are synchronized to the channel DB).

Example request with Accept-Language header:

This header sets RequestContext.Language:

It reads the Accept-Language header
Takes the first language entry
If the header is missing or null, Commerce CSU sets RequestContext.Language as per the channel's (online store's) default language

Example 1: Country/region names translated in shipping APIs

If you maintain country/region translations in D365 (Address setup > Country/region > Translations), CSU can return the translated short/long names when you pass Accept-Language.

API example

GET /Commerce/GetCountryRegionsForShipping

Behavior

With Accept-Language: the response returns country/region ShortName and LongName in German (if translations exist).

This is a clean way to avoid translation work in your frontend.

Example 2: Mode of delivery descriptions translated (with a gotcha)

Mode of delivery: you cannot translate the code, but you can translate the description. CSU applies those translations when Accept-Language is set.

Works for:

GET /Commerce/GetDeliveryOptions
GET /Commerce/Carts('{Retail_CartId}')/GetDeliveryOptions

Does NOT work for

GET /Commerce/OrgUnits/GetChannelDeliveryOptions

Gotcha (important): If you configure Mode of delivery translations in D365, define translations for all languages, including the default online store language. Otherwise, when the Accept-Language header is missing and CSU will fall back to the default language (which is missing a translation), CSU will pick the “first available translation record” in an unexpected language.

Example 3: Product search and Accept-Language (subtle but useful)

This is where things get interesting. CSU product search isn’t “one engine”. Different endpoints behave differently with language:

Products/Search and SearchByText do not support searching in another language than the channel default.
SearchByCriteria can search in a non-default channel language if set in the request body. Note: this language needs to be listed under languages on the online store setup in D365 ERP to make this work.
With Accept-Language on SearchByCriteria, attribute/search condition evaluation runs against the online store's default language and the Accept-Language language (if different), while Product Name/Description are returned in the Accept-Language language.

Concrete example For a channel with default language en-us, using:

POST /Commerce/Products/SearchByCriteria
Accept-Language: de
Search condition: "Brush" (default language) or "Bürste" (Accept-Language) — both work

Result: both search conditions return the same product, with Name and Description in German — "Bürste-CB 4-reihig Kunststoffkopf 600". Attribute values will also be translated into the Accept-Language (de) if translations exist.

Why I’m sharing this

Because this is the kind of thing that:

improves customer experience immediately,
reduces frontend translation complexity,
and is easy to miss if you only test with default channel language.

If you’re building headless Commerce on CSU, add Accept-Language to your test scripts and see where it changes your responses.

Commerce CSU V9.56/10.0.46 - January 2026

Fri, 16 Jan 2026 00:00:00 GMT

For anyone new in this field: D365 Commerce CSU is shipped as part of D365 ERP. It comes with 650 APIs out of the box which enable the use of ERP and CRM data and business logic in B2C and B2B commerce.

I have been using Commerce CSU for integration scenarios since its early days around 2015. I am sharing my knowledge to enable anyone to use Commerce CSU effectively for their B2C and B2B Commerce applications (e-commerce, POS, Customer Service etc.)

In that context, I am publishing these unofficial D365 Commerce CSU Release Notes with any upcoming new Dynamics 365 ERP and Commerce platform Release.

A-Release Note Details

These release notes are applicable to the following CSU and D365 ERP version:

This release (9.56 / 10.0.46) is a clear step toward more modern, resilient checkout and payment experiences—especially for headless and eCommerce scenarios where payment completion doesn’t always happen instantly.

The biggest shift is the platform maturing around asynchronous payments: think Pay-by-Link, redirect-based flows, and webhook-driven confirmations. The data model now carries clearer indicators of what has been paid, what is still pending, and how payment state should be interpreted during checkout—making it easier to build reliable customer journeys even when the payment outcome arrives later.

Alongside that, there’s a noticeable emphasis on operational robustness. Cart processing gains stronger concurrency safeguards to reduce “double actions” and conflicting updates during payment-in-flight situations—exactly the kind of hardening that matters in high-volume retail and self-checkout patterns.

There are also improvements aimed at traceability and reconciliation in payment-adjacent flows, including richer reference data for third-party gift card scenarios, which helps with auditing, customer support, and recovery when external providers are involved.

Finally, the release includes smaller but meaningful enhancements for store operations efficiency, reducing friction in staff-related workflows and improving performance where bulk lookups are common.

Overall: if you’re building or extending Commerce experiences beyond the standard storefront—especially headless—this version is worth attention because it strengthens the foundation for payment modernization, better observability, and fewer edge-case failures at checkout.

B-Downloads for your convenience

Latest V9.56 CSU full interface specification in OpenAPI/Swagger format: Link
Lestest V9.56 Overview of required Commerce Roles for all nearly 800 APIs: Link

C-API and Schema Changes compared to the previous version 9.55/V10.0.45

C.1 New APIs

C.2 Removed APIs

None

C.3 Data Model Changes

1) Asynchronous payments

What changed:

Cart and SalesOrder payloads gained async-payment-aware totals/flags: AmountPaidWithAsynchronousPayments, IsRequiredAmountPaidInclAsynchronousPayments)
AsynchronousPayments[] gained new tracking/typing fields (AsynchronousPaymentStatusValue, ExtensibleAsynchronousPaymentTypeValue, ServiceAccountId) plus a default change (AsynchronousPaymentTypeValue 1 → null).

Where it shows up:

POST /Carts/CreateAsynchronousPayment
GET /SalesOrders/GetSalesTransactionCheckoutResultByTransactionId
Many Cart-returning operations now include the same fields

Likely intent. This is the API surface for asynchronous payment support for headless and eCommerce: initiate an async payment (CreateAsynchronousPayment), handle notifications, and track checkout status to completion (GetSalesTransactionCheckoutResultByTransactionId). This is part of the pay-by-link / delayed completion scenarios and automatic cancellation if payment isn’t completed.

2) External gift cards

What changed: Line-level ThirdPartyGiftCardInfo gained TransactionReferenceData and the change appears on CartLine shapes across many cart mutation/read APIs (and any sales-line/order-history shapes that include the same object).

Where it shows up:

POST /Carts({id})/AddPreprocessedTenderLine
POST/Carts({id})/UpdateCartLines
Any Cart-returning operation that returns CartLines repeats the same block

Likely intent: supports external gift cards (provider-integrated) alongside internal gift cards. A new TransactionReferenceData field fits the need to carry provider reference/trace/audit data for reconciliation, customer support, reversals, retries, and “prove what happened” scenarios typical of third-party gift card flows.

3) Payments connector extensibility

What changed: PaymentMerchantInformation gained ConnectorCustomSettings, repeated across endpoints that return Cart.MerchantProperties[] (again, many cart APIs).

Where it shows up: Appears alongside other Cart schema changes in cart-returning endpoints, e.g.

POST /Carts({id})/OverrideCartLinePrice
POST/Carts({id})/OverrideCharge

Likely intent: A generic “custom settings” bag is a common pattern to expose connector-specific configuration without forcing new first-class fields per PSP/provider.

C.4 New flight keys

Microsoft implements new CSU flight keys with every new CSU version. These undocumented flight keys steer specific CSU behavior. Here is a list of the latest keys for V9.56/V10.0.46

Based on our analysis of the code, the flight keys appear to be used as follows:

Disclaimer

For authoritative product information, consult Microsoft's official documentation at learn.microsoft.com.

How can we optimize performance of D365 APIs for use in headless environments?

Thu, 08 Jan 2026 00:00:00 GMT

D365 F&O APIs and Dataverse APIs are usually not a great fit for headless environments. Not because they don’t work — but because their performance is often not stable or predictable enough for real-time use cases like:

Webshops
Customer service applications
Custom frontends
Mobile apps

In many scenarios, response times are simply too slow — or worse — inconsistent. Ironically, this affects not only custom-built apps, but also Microsoft’s own solutions that rely on these APIs, such as D365 e-commerce 🤔.

“But Commerce APIs are designed for this… right?”

Yes — Commerce APIs are the exception.

They are built for:

high-frequency calls
carts, pricing, checkout
POS and online scenarios

But even Commerce APIs can suffer from a familiar problem: ⚠️ After a few minutes of inactivity, performance can suddenly drop — sometimes becoming 3× slower.

Welcome to the world of cold APIs.

⚠️The real challenge: SaaS means limited control

Here’s the uncomfortable truth:

D365 ERP, Dataverse, and Commerce are SaaS platforms
We don’t control the infrastructure
We can barely influence allocated resources
We cannot “just tune the server”

And then there’s an even bigger issue…

Many APIs don’t just read data — they do things:

Create orders
Update addresses

We don’t want to pollute the database just to keep an API warm.

So the question becomes:

✅ How do you keep D365 APIs warm even if they actually mutate data?

🔑Suggested approach for D365 Finance & SCM (ERP)

❌ Avoid using D365 ERP APIs for headless scenarios whenever possible.

✅ Preferred alternatives:

Use Commerce CSU APIs, or
Use a custom data cache with a custom API layer that periodically synchronizes with the D365 ERP database (conceptually, this is exactly what Commerce CSU already is — although it contains many APIs with complex business logic).

📊 Before go-live:

Ensure sufficient telemetry and evidence is in place to properly scale a D365 ERP environment for integrations - This is a fundamental condition for API performance.
To achieve this, it is strongly recommended to simulate real-life business processes in a Tier-4 / performance test environment

🚦 Prevent throttling and rate-limit issues at all cost**:**

Carefully distribute data loads across different resources in advance
Use tools such as Azure API Management (APIM) to shape and control traffic

⚙️ If D365 ERP APIs are still required, and the baseline scaling prerequisites have been met by Microsoft, follow the strategy below to maximize API performance. OData APIs and Custom APIs, both require a different approach (see picture below):

✅ For OData end points:

Enable the OData metadata cache at AOS startup (introduced with PU32) to avoid cold starts to build metadata cache.
Raise throttling priorities to ensure sufficient resource allocation.
To keep GET requests warm at deeper levels (Data and Business Logic), leverage periodic requests. Common pitfall here is that a node may be warmed up which is not used for business critical API requests later, so optimize frequency of the "keep warm" requests to avoid this.
To keep PATCH/POST requests warm at deeper levels (Data and Business Logic), create a dedicated WarmUp endpoint if data access and business logic routes are different than the GET operation for the respective Data Entity.

✅ For Custom end points:

To keep the caching level warm, Create a WarmUp endpoint to warm up the Service Group (JIT, metadata, caches).
Raise throttling priorities to ensure sufficient resource allocation.
If API performance is not sufficient yet, enhance the "WarmUp" endpoint by implementing code to trigger query paths and business logic which is triggered by the actual endpoint as well. Introduce roll back or other patterns to stay as close to the real paths without creating actual data.
As mentioned before common pitfall with periodc "keep warm" requests is that a node may be warmed up which is not used for business critical API requests later, so optimize frequency of the "keep warm" requests to avoid this.

🔑 Suggested approach for Dataverse: a very different story

Dataverse behaves fundamentally differently from D365 ERP:

👉 Metadata is platform-wide cached
👉 First calls are rarely significantly slower
👉 Performance is often better, but also less predictable
👉 Conceptually, there is no real difference between OData and “Custom APIs”
👉 Stricter API limits apply
👉 “Keep-warm” calls often hurt more than they help
👉 No concept of service group warm-up like in ERP

Conclusion: More “SaaS” than D365 ERP, making performance stability harder to enforce and guarantee.

Recommended Dataverse strategy

1️⃣ Make plugins cold-start friendly
2️⃣ Prefer asynchronous patterns
3️⃣ If performance stability is critical:

👉 Introduce a custom staging database (e.g. Cosmos DB) + Expose via a custom API layer on top

🔑 Suggested approach for Commerce CSU

Commerce APIs are already optimized for headless usage — but tuning may still be required.

Best practices

Scale out with multiple scale units for large implementations (cost increase is often surprisingly modest: ~12–30%)
Keep Commerce data lean and efficient: avoid the use of catalogs, deeply nested large category structures and unnecessary assortment sizes.
Prefer IV (Inventory Visibility) over channel-side inventory sync, especially in case of large stock files.
Reduce synchronous RTS usage via parameterization (use channel db to HQ synch patterns instead)

Warming strategies

Slow GETs → one periodic "keep warm" request every ~10 minutes is often enough
Many nodes? Similar issue as we saw with D365 ERP → Increase frequency to 2–5 minutes
Slow POSTs? 👉 Last resort: custom warm-up endpoints that touch deeper CRT and data layers used by the real operation

🔑 Your low-code helper: the “keep-warm” Logic App

If the performance of your D365 ERP and Commerce APIs can be sufficiently improved by keeping the API warm through periodically triggering an endpoint—either directly or indirectly via a “WarmUp” endpoint—it can be very useful to implement this using a Logic App.

First, because it is an easy-to-maintain and easy-to-understand low-code solution, and second, because it is simple to configure:

1️⃣ Recurrence can be configured directly in the Logic App

2️⃣ If the Logic App interacts with endpoints that are abstracted through Azure API Management, you can avoid using service connections altogether — making the Logic App easy to deploy and move between environments

3️⃣ Sample payloads can be maintained externally, for example in files or storage accounts, and reused by the Logic App (effectively decoupling data from logic)

Below is an example of such a Logic App:

Final thought — and an open question

I’m convinced many people in the D365 community have great ideas and best practices around this topic.

👉 How do you handle API performance in headless scenarios? 👉 What patterns have worked — or failed — for you?

Let’s share experiences.

Is there an API-first alternative for Dual Write?

Thu, 08 Jan 2026 00:00:00 GMT

This is a question which has been bugging me for the last three years.

Every time I read a Microsoft Learn article article about D365 SCM or D365 CE being the price master (example), I catch myself thinking halfway:

“Really? Isn’t there an easier way?”

Here’s what we’re doing today:

1️⃣ The process (e.g. quote-to-order) runs in system A – D365 CE
2️⃣ System A asks system B – D365 SCM to recalculate pricing
3️⃣ The recalculation happens in B
4️⃣ System B syncs the data back to A
5️⃣ The process continues… hopefully

But what if something fails halfway?

⚠️ What if the recalculation (step 3) throws an error?

⚠️ Or the data sync (step 4) doesn’t bring back all price components — discounts, attribute pricing, shipping charges, minimum-order fees, overrides?

Couldn’t we do this faster, cleaner, and more reliably?

✅ My suggestion: API-first.

Let CE call SCM’s pricing engine directly in real time — no sync, no waiting, no partial data. You get a single, complete response containing every price component. Instant feedback, fewer moving parts, and a process that never pauses “mid-flow.”. But that’s theory? Can it work in reality?

An API-First Approach

Let’s explore the approach: there’s no data sync between D365 CE and D365 ERP. Instead, quote or order lines in CE are sent to an API endpoint that returns a detailed pricing breakdown — typically within 500–750 ms ⚡.

CE can trigger this in multiple ways:

as an in-process (background) flow, automatically invoked when a quote line is added, updated, or deleted,
or more explicitly via a JavaScript-enabled “Recalculate Pricing” button.

But the real question is… 👉 What’s behind that API endpoint?

D365 SCM OData and custom APIs aren’t known for stable, high-volume performance. They work fine for batch integrations, but for consistent real-time request/response patterns, they often fall short. There are many reasons — complex data models, non-optimized joins, limited caching — but the bottom line is: ⚠️ it’s very hard to make them truly performant consistently, especially under high volume.

By contrast, the D365 Commerce APIs are built for speed and consistency. They’re optimized for high-frequency scenarios like online cart operations and POS transactions, where performance must be predictable.

Even better: the Commerce Cart APIs support both the existing SCM pricing engine and the new Unified Pricing model — built on the same Commerce pricing foundation, now enhanced with attribute-based pricing.

SCM and Unified Pricing API for Quote to Order

So, to explore the options, I decided to develop a single API endpoint, implemented via an Azure Logic App, which wraps several existing D365 Commerce APIs into one cohesive call.

Create Cart
Add Cart Line
Delete Cart

The API requires a very simple request body:

But it returns full pricing, including gross pricing, net pricing, shipping charges, minimum order fee, taxes, customers discounts, attribute based discounts etc. – Just a snippet:

No data sync. No duplication of logic. Just a question and a fast, detailed answer — so the Quote-to-Order process can simply continue. ✅ No process breaks. ✅ No endless loading spinners.

But wait… doesn’t Commerce require a Channel and an OUN (Operating Unit Number) to function? Can it really work with just a customer and quote lines as input? 🤔

How to “hide” the Commerce channel concept for Dataverse

By implementing a very simple Default flag on the channel in the Customer hierarchy form in D365 SCM, we can relate a B2B customer to 1 default channel in D365 SCM:

We can then accept the B2B customer as header parameter in the request from Dataverse, as opposed to the OUN (=Commerce channel):

Benefits

In my view, this API-first approach offers significant advantages over Dual Write, especially for the “Better Together” stories between D365 Dataverse (CE, Sales, Field Service, IOM, etc.) and D365 SCM — where large data volumes and complex business logic come into play.

If you count how many data entities need to sync through Dual Write just to make advanced pricing details flow between the two ecosystems… you’d quickly see how fragile and maintenance-heavy that becomes.

To make it concrete, the table below summarizes the most relevant benefits of API-first vs. Dual Write for this scenario 👇

Further optimization

The Logic App which I pragmatically built for my prototype is a so called Composite API: it wraps multiple standard Commerce APIs into 1 endpoint. There are platforms which do this wrapping more efficiently at run time than a Logic App. An example is FastAPI.

Another optimization would be to implement api-key authorization between Dataverse and D365 SCM. The ideal platform for this is Azure API Management. APIM can abstract Commerce CSU App or Customer based authorization and work securely with api-key towards Dataverse for simplification (which is easier to implement between the systems).

If you’re interested in this API-first approach, feel free to get in touch via patrickmouwen.com/contact. I specialise in API-driven integration between D365 SCM, Dataverse, Commerce and third-party systems.

Commerce CSU V9.55/10.0.45 - September 2025

Wed, 17 Sep 2025 00:00:00 GMT

For anyone new in this field: D365 Commerce CSU is shipped as part of D365 ERP. It comes with 650 APIs out of the box which enables the use of ERP and CRM data and business logic in B2C and B2B commerce.

In that context, I am publishing these unofficial D365 Commerce CSU Release Notes with any upcoming new Dynamics 365 ERP and Commerce platform Release.

A-Release Note Details

These release notes are applicable to the following CSU and D365 ERP version:

This new version (9.55.25240.5 / 10.0.45) doubles down on payments and checkout hardening— Changes pivot around five themes:

External/third-party gift cards after payment: new line-level fields on (e.g., approval and tender type) show up across /Carts, /SalesOrders, and related reads—enabling the “issue/fund only after payment succeeds” design that’s critical for SCO and fraud control.
Guest checkout control via API: /Carts/GetConfiguration now reflects the Allow anonymous checkout switch, aligning headless flows with your Online Functionality Profile.
Payment notification clarity: /PaymentNotifications surfaces a normalized processing status, making webhook-driven flows (including Pay by link) easier to reconcile and automate.
Refund usability: expanded EmployeePermissions support showing all refund methods (when policy allows), reducing manager overrides in return flows.
Azure Maps at channel level: /OrgUnits/GetOrgUnitConfiguration adds flags/keys so Store Locator, address suggest, and POS scenarios can move off Bing Maps (discontinued) cleanly.

Pay by link: if you’re using the out-of-box Adyen connector, these API updates complement that journey: notifications carry clearer status for Dataverse-backed processing, checkout settings are enforced consistently across headless vs. storefront, and post-payment gift-card issuance avoids value creation before funds clear.

B-Downloads for your convenience

Latest V9.55 CSU full interface specification in OpenAPI/Swagger format: Link
Lestest V9.55 Overview of required Commerce Roles for all nearly 800 APIs: Link

C-API and Schema Changes compared to the previous version 9.54/V10.0.44

C.1 New APIs

None

C.2 Removed APIs

None

C.3 Data Model Changes

1) External gift cards: new flags on line-level ThirdPartyGiftCardInfo

What changed (multiple APIs): gained and in cart/sales-line schemas. Likely intent: enable the “fund gift cards after payment is completed” design and surface the card’s tender type + approval state through the API (important for POS/SCO flows and fraud/risk controls).

2) Guest checkout hardening: CartConfiguration.AllowAnonymousCheckout

What changed: adds (returned by ). Likely intent: close the loophole where guest checkout could be invoked via APIs even if the storefront didn’t show it—making the API honor the Online Functionality Profile setting.

3) Refund policy usability: EmployeePermissions.AllowAllRefundPaymMethods

What changed: adds across Employees endpoints (, , and ). Likely intent: if a channel return policy hides certain payment methods and “Manager override” is on, authorized non-managers can still “Show all refund options” and finish the return—removing the manager dependency. This landed with 10.0.44 but is now reflected in the API surface.

4) Payment notifications: ProcessingStatusValue

What changed: adds to the entity and query shapes; / enums extended to include it; CRUD and updated accordingly. Likely intent: expose a normalized processing status on payment notifications for sorting/filtering and operational reconciliation (e.g., async gateway callbacks or retries).

5) Azure Maps enablement at channel level: ChannelConfiguration.AzureMapsEnabled/AzureMapsApiKey

What changed: extends with and . Likely intent: support Bing Maps retirement by allowing Azure Maps for Store Locator, BOPIS, address autosuggest, etc., and for POS scenarios.

C.4 New flight keys

Microsoft implements new CSU flight keys with every new CSU version. These undocumented flight keys steer specific CSU behavior. Here is a list of the latest keys for V9.55/V10.0.45.

Based on our analysis of the code, the flight keys appear to be used as follows:

Disclaimer

For authoritative product information, consult Microsoft's official documentation at learn.microsoft.com.

🧩 Unlocking True Headless B2B Commerce with D365 — Without Customizing the Core

Fri, 27 Jun 2025 00:00:00 GMT

Dynamics 365 Commerce has matured into a strong platform for both B2C and B2B scenarios. Yet for organizations aiming to build a headless B2B strategy, there’s a fundamental limitation: the data model is rooted in individual customer accounts. For modern B2B journeys involving multiple organizations and users, this creates real challenges.

At 365Connect, we’ve faced this first-hand. Instead of fighting against the platform, we developed an approach that:

⚡ Works without core customizations to D365 Commerce
📦 Keeps the standard Commerce APIs fully intact
🌍 Enables modern B2B use cases

🚧 The Core Challenge: Identity vs. Organization

In the out-of-the-box D365 Commerce model, every user is tied to an individual customer account. Transactions are recorded on that account, with the B2B customer only referenced as an invoice account.

This means:

Collaboration within the same B2B organization is difficult
Transacting on behalf of different legal entities is complex
Headless orchestration between frontends and ERP is limited

💡 Our Approach

The current person customer/B2B customer data model is a challenge here. I've seen heavy customizations in the market to change the model to contact- and B2B organisation-based which hits the core foundation of the Customer context in D365 CSU.

We decided to leverage a creative solution which does not impact core CSU functionality.

The solution capitalizes on having 1 enterprise identity account (=person customer) per B2B organization which is shared across the identities (=B2B users). This 1:N association is realized in the standard RetailExternalIdentityToCustomerMap table.

By developing a custom CSU endpoint (/Commerce/TSFCActivateOrganization) working against the RetailExternalIdentityToCustomerMap, the online shop frontend is in control to bind a logged-in user (identity) to another organization from the online shop's front end. This is truly headless and unlocks true Order on behalf for B2B organizations where 1 buyer can now work on behalf of multiple organizations (like subsidiaries of the same organization).

One difficult hurdle we faced here was the fact that the customer context is cached for 5 minutes (hard coded in an area which cannot be extended). So, even though the logged-in user had switched to a different organization, the customer context still continued to use the original context for 5 minutes. We resolved this by implementing a request handler for the GetCommerceIdentityByExternalIdentityServiceRequest and override it with our own custom caching.

Upon standard sales transaction to order conversion in D365 ERP, the person customer account is swapped by the actual B2B account (by look up via the Customer Hierarchy) before the order is created.

This end-to-end works very well and covers all beforementioned B2B requirements without hitting any core CSU code. As such, this solution component, together with a few other tweaks, is a game changer for using D365 Commerce CSU in the B2B market.

✔ Zero custom code in the core ✔ ERP receives clean and correct customer assignments ✔ Headless architecture stays lean and flexible ✔ Works with Microsoft D365 frontends as well as third-party apps

🎯 Why This Matters for B2B Commerce Leaders

As more enterprises embrace composable commerce and API-first architectures, the last thing they want is heavy customization in their core systems.

With our approach, B2B leaders can:

👥 Empower employees to place orders on behalf of multiple organizations
🛒 Enable collaboration through shared carts
🔗 Support dealers and partners in B2B2B models
📈 Accelerate innovation while keeping upgrade paths safe

🤝 An Open Invitation to Microsoft

We see Microsoft investing heavily in headless, composable, and AI-driven Commerce. Our solution shows that even foundational gaps can be addressed — by understanding the platform deeply and applying it creatively rather than customizing it.

We invite Microsoft and partners to join us in scaling this approach, and making D365 Commerce the most adaptable platform for headless B2B commerce in the market.

🧭 Let’s Talk

Are you a B2B commerce leader, Microsoft architect, or ISV struggling with these challenges?

🔗 Get in touch via patrickmouwen.com/contact. Happy to help you unlock the full potential of headless B2B Commerce with D365 — without unnecessary complexity.

Commerce CSU V9.54/10.0.44 - June 2025

Thu, 26 Jun 2025 00:00:00 GMT

For anyone new in this field: D365 Commerce CSU is shipped as part of D365 ERP and comes with 650 APIs out of the box for use in B2C and B2B commerce.

I have been using Commerce CSU for integration scenarios since its early days around 2015. I am sharing my knowledge to enable anyone to use Commerce CSU effectively for their B2C and B2B Commerce integration scenarios (e-commerce, POS, Customer Service etc.)

In that context, I am publishing these unofficial D365 Commerce CSU Release Notes with any upcoming new Dynamics 365 ERP and Commerce platform Release.

A-Release Note Details

These release notes are applicable to the following CSU and D365 ERP versions:

This CSU API release seems to focus on the latest Pay by link feature, which is availablefor the out-of-box Adyen payment connector. See more details here on Microsoft learn.

The Pay by link feature is based on the following components:

Payment notification service which is triggered from an Adyen webhook, sent to a Dataverse endpoint
The Dataverse endpoint is set up in the Commerce shared parameters
Features "Payment notifications", "Pay by link payment", "Enhanced wallet support" and "Payment improvements" to be enabled in D365 ERP
APIs which handle Payment notifications and Payment link
Pay by link button in the POS

B-Downloads for your convenience

Latest V9.54 CSU full interface specification in OpenAPI/Swagger format: Link
Lestest V9.54 Overview of required Commerce Roles for all nearly 800 APIs: Link

C-API and Schema Changes compared to the previous version 9.53/V10.0.43

C.1 New APIs

Related to Payment notifications:

For creating and maintaining a payment link:

C.2 Removed APIs

None

C.3 Data Model Changes

New property AsynchronousPayments on API requests and responses which work against Carts and SalesOrders
New property LoyaltyGroup]/LoyaltyProgramNextTierStatus on POST /Commerce/GetCustomerLoyaltyCards, /Commerce/ScanResults APIs and POST /Commerce/GetLoyaltyCard
New property IsUnifiedPaymentExperienceDisabled on GET /Commerce/GetDeviceConfiguration()
New property CardInteractionConfigurations on GET /Commerce/GetTenderTypes()

C.4 New flight keys

Microsoft implements new CSU flight keys with every new CSU version. These undocumented flight keys steer specific CSU behavior. Here is a list of the latest keys:

Disclaimer

For authoritative product information, consult Microsoft's official documentation at learn.microsoft.com.

Commerce CSU V9.53/10.0.43 - March 2025

Thu, 20 Mar 2025 00:00:00 GMT

For anyone new in this field: Commerce CSU is shipped as part of D365 ERP and can seamlessly integrate any Professional App, Social commerce, e-commerce or POS solution with D365 ERP.

I have been using Commerce CSU for integration scenarios since its early days around 2015. My mission is to empower anyone to use Commerce CSU effectively for their integration scenarios. Therefore, I will publish these unofficial CSU Release Notes with any upcoming new Dynamics 365 ERP and Commerce platform Release.

A-Release Note Details

These These release notes apply to the following CSU and D365 ERP version:

B-Downloads for your convenience

Latest V9.53 CSU full interface specification in OpenAPI/Swagger format: Link
Lestest V9.53 Overview of required Commerce Roles for all nearly 800 APIs: Link

C-For the Techies: API and Model Changes compared to the previous version 9.52/V10.0.42

C.1 New APIs

There's one in the works, but not fully implemented yet:

This API seems to control a stored procedure to ensure post-processing for specific tables after the CDX synch between HQ and CSU DB has run.

For example, if tables like ECORESPRODUCTCATEGORY or RETAILPRODUCTATTRIBUTESLOOKUP are affected by the data synch between HQ and CSU database, then return policies are updated.

C.2 Removed APIs

None

C.3 Data Model Changes

Property AllowedMinimumDaysForRequestDelivery is added to Microsoft.Dynamics.Commerce.Runtime.DataModel.CartConfiguration
Property AllowedMaximumDaysForRequestDelivery is added to Microsoft.Dynamics.Commerce.Runtime.DataModel.CartConfiguration.

These properties are maintained in the Online store functionality profile in D365 ERP:

C.4 New flight keys

Microsoft implements new CSU flight keys with every new CSU version. These undocumented flight keys steer specific CSU behavior. Here is a list of the latest keys:

D-Sources

With my Team, I have developed a couple of tools to collect the information above for you:

Automation to extract the OpenAPI specification from the latest Commerce CSU package
Automation to compare the OpenAPI specification with a previous OpenAPI specification and identify any changes
Automation to parse the latest DLL with D365 Commerce OData end points and extract role identification from the DLL:

E-About the Author

I've been integrating non-Microsoft e-commerce solutions with D365 ERP and CRM since the early days of AX for Retail in 2009. Already back in 2015, I connected an E-commerce platform to D365 ERP by a predecessor of the current D365 Commerce CSU for the first time.

Get in touch via patrickmouwen.com/contact if you need help in implementing:

D365 Commerce on top of ERP including D365 Store Commerce App and D365 e-commerce
Customizing/Extending D365 Commerce, CSU, Store Commerce App and e-commerce
B2C and B2B Headless Commerce/D365 ERP integrations via Commerce CSU
B2C and B2B Commerce Platform/D365 ERP integrations via Commerce CSU
Customer Service operation based on D365 ERP and D365 CE data utilizing Commerce CSU
Social Commerce/D365 ERP integrations via Commerce CSU
Professional App (React.JS)/D365 ERP integrations utilizing Commerce CSU
POS/ERP integration

Disclaimer

For authoritative product information, consult Microsoft's official documentation at learn.microsoft.com.

Case #3: Why using Commerce CSU for Headless Commerce beats custom-built Solutions

Fri, 14 Mar 2025 00:00:00 GMT

In this series of blog posts I'll share insights, strategies, and proven solutions to address the most common roadblocks and challenges in connecting non-Microsoft e-commerce to D365 ERP.

Today's Case

Introduction

In this blog post, we'll take a look at the benefits of using Commerce CSU to expose D365 ERP data for Headless e-commerce solutions. We'll do this by comparing with more traditional custom approaches.

To take an honest approach, we must first distinct the various types of e-commerce solutions:

Traditional Commerce Platforms: In traditional Commerce platforms, the frontend (user interface) and backend (server, database) are tightly integrated. Backend systems like D365 ERP often deliver data to the platform by periodic data sync and receive transactions via periodic data sync.
Headless Commerce: Headless commerce decouples the UX frontend from the backend. Backend systems like D365 ERP could deliver their product, pricing, attribute, payment, order history and other data by direct API connectivity if performance allows that.
Hybrid: combines elements of both traditional and headless architectures. Businesses might use a traditional platform for core e-commerce functionalities while integrating headless solutions for specific channels or services. Backend systems like D365 ERP could deliver their data by direct API connectivity, data synch or a combination.

Traditionally, companies implemented more monolithic commerce platforms to benefit from its 'all-in-one' nature. The platforms were connected to backend systems like D365 ERP and CRM by data synch.

Over the last 5 years, the benefits of more monolith platforms have been challenged fundamentally by the following trends:

Shift in Consumer behavior: Shoppers demand seamless omnichannel experiences, AI-driven personalization, and flexible business models (subscriptions, marketplaces), which platform monoliths struggle to support.
Digital Touchpoints – A traditional website is no longer spearheading sales. Sales is happening via a variety of digital touch points like mobile apps, social commerce (TikTok, Instagram etc), marketplaces, AI chatbots and agents and smart devices (like fridges which can re-order). This requires a headless architecture which enables brands to serve all these touchpoints from a single backend by exposing APIs, whereas traditional commerce platforms struggle because they were built for a single-channel experience (usually web-based).
AI & Automation Require API-First Integration – AI-powered search, recommendations, and automated workflows need real-time data access, which rigid, monolithic architectures cannot efficiently provide.
Regulatory & Security Pressures Have Increased – Compliance with GDPR, CCPA, and PSD2 is complex, and traditional platforms require costly updates to stay compliant, whereas headless solutions can lean on backend systems like D365 ERP which are already compliant.

Because of these trends, headless commerce is quickly winning ground over traditional commerce platforms, both in B2C and B2B.

The role of D365 Commerce CSU

I've been working in the domain of D365 Business Applications and Commerce for more than 15 years. If I take an honest look at this domain I can only conclude that D365 ERP and Dataverse (baseline for D365 CE, Customer Service etc) are not suitable to serve Headless Commerce solutions. This is exactly the reason why so many partners and customers have developed a wide variety of custom services and solutions to integrate D365 ERP and CRM/Customer Insights with e-commerce solutions.

In most cases, these efforts hit hard walls like:

Excessive CAPEX investment: most available D365 ERP and Dataverse APIs are Create, Read, Update, Delete (CRUD), but headless commerce requires business-process APIs for areas like Cart management (pricing, promotions, taxes, shipping), Real-time inventory availability (e.g., ATP calculations, warehouse fulfillment) and AI-driven product search and recommendations.
Poor performance: standard D365 ERP OData APIs and Dataverse APIs are not designed for high-throughput, low-latency e-commerce transactions. APIs are often batch-oriented, making them inefficient for real-time customer interactions (e.g., live cart updates, checkout). Dataverse throttling and API rate limits can block high-volume requests, making it unsuitable for peak traffic.
Lack of omnichannel focus: the solution if often architected to serve only a single channel or digital touchpoint.

Dynamics 365 includes a built-in headless architecture that enables direct use of Dataverse and D365 ERP data in headless frontends. This architecture is designed to support own D365 e-commerce and D365 Store Commerce (POS) solutions, making it a robust and well-suited backend for any headless commerce frontend.

This picture outlines the architecture which is part of the standard D365 offering:

1. Dual Write: Data can be synched bidirectionally between Dataverse and D365 ERP in real time by using Dual Write. D365 ERP comes with D365 headless commerce engine (CSU) which consists of a database, microservices and OData API layer with 650 APIs.

2. D365 headless commerce engine: Synching data from Dataverse (via Dual Write) and D365 ERP to the CSU database is out-of-box available bi-directional and near-real time (usually with synch intervals between 1 and 3 minutes**).**

3. Headless commerce architecture: The CSU database acts as a local data cache. It is also possible to deploy multiple CSUs which keep latency to the digital touch points low. Operating on top of the local data cache and microservices architecture, the 650 APIs are performant enough to directly serve as backend for headless commerce. The APIs include commerce specific APIs which are not available in D365 ERP or Dataverse, but which operate directly on top of master data, setup and parameters in D365 ERP and Dataverse. Examples are Cart APIs and APIs which expose Unified Pricing.

4. Additional services. Dataverse Apps and D365 ERP struggle to provide a performant search experience across Inventory, Products and Customers. D365 Commerce headless engine resolves this by teaming up with Azure AI powered search. For example, D365 ERP Product and Attribute information is indexed in Azure AI Search (formerly known as Cognitive search) which is utilized by the Commerce Product Search APIs directly. D365 Commerce CSU also integrates with Customer Insights utilizing Microsoft Copilot and Inventory Visibility.

5. Ready-for-use integration.

The 650 APIs provided by the D365 headless commerce engine (CSU) are ready-for-use. Here on my website you can find a list of all available APIs presented in Swagger UI.

Preliminary conclusions

An article like this can never outline all arguments why D365 commerce headless engine is such a treasure for quick development of high performant and rich frontends for any digital commerce touch point in headless architectires.

But by highlighting just a few benefits compared to custom integration solutions based on D365 ERP and Dataverse, I hope I could convey my personal experience that the use of d365 commerce headless engine will always beat any other custom solution. Whether looking from the angle of efficient investment, time-to-market, omnichannel strategy, scalability or performance, D365 commerce headless engine is always a better alternative than a custom solution.

However, all of these benefits would all be wiped out in one sweep if D365 headless commere engine would not be affordable.

So, before jumping into final conclusions, let's first address this often-heard argument against the use of D365 Commerce headless engine.

Is D365 Commerce CSU affordable?

To answer this question, let's first align on some fundamental assumptions:

ERP Backend licensing. Deploying Commerce CSUs requires an organization to have 20 Commerce backend licenses. Standard ERP (Finance/Supply Chain) licenses can often be swapped for 20 Commerce licenses against no additional cost.
Headless licensing and headless use. Commerce CSU was originally developed for D365 e-commerce and D365 Store Commerce App. So, license-wise you either pay for deploying x number of devices running Store Commerce App or D365 e-commerce. If you want to use the D365 headless commerce engine (CSU) for headless frontends, the licensing does not map neatly onto either model, so it becomes a commercial conversation rather than a list price. Treat it as negotiable and get it costed early: it is an architectural assumption with a material price attached, and it should be tested before it becomes a commitment. Note that this licence cost grants access to your ERP and Dataverse data for hundreds or thousands of customers.
Non-headless use. Integrating D365 ERP and Dataverse with more traditional monolith platforms is a totally different ball game than headless architectures. The platforms often come with significant investment due to the proclaimed "all-in-one" nature and integrations are also relatively costly as they involve asynchronous data flows and often complex data mappings and custom business logic (for example for pricing and inventory calculations). Due to this high investment level for platform and integrations, there is often no budget left for d365 headless commerce engine (CSU). Therefore, this article only focuses on the cost for headless e-commerce architectures, which many companies in B2C and B2B commerce transition to at the moment.
Affordable cost per Order (CPO): Total cost of an order including operational and IT related OPEX is often measured as % of Average Order Value (AOV). A portion of the CPO/AOV % will be contributed to OPEX for maintaining the headless commerce solution (frontend and backend). It's always arbitrary, but from what I've seen in my projects over the last 10 years, an affordable CPO/AOV % for a B2C front- and backend is about 0.5-1% for B2C (as AOV and margins are lower) and 1-2% for B2B. If we split the cost for frontend and backend functions in two halfs, then an affordable CPO for backend integration would be about 0.2-0.5% for B2C and 0.5-1% for B2B.
AOV ranges: B2C businesses usually operate in lower Average Order Value (AOV) ranges than B2B organizations. B2C businesses mostly stay within USD 50-250 range where B2B AOVs are often between USD 500 and USD 2000.

Now, with this in mind, I've used AI tools to calculate different price points based on the licensing model for D365 headless commerce engine (CSU) - See licensing guide March 2025.

I calculated the price points by the following assumptions:

AOV for B2B in USD: 49 - 100 - 250 - 750 - 1000 - 1500 - 2500 - 4000 - 7500
AOV for B2C in USD: 25 - 50 - 100 - 150 - 250 - 750 - 1000 - 1500 - 2500
Transactional volume for B2B per year: 2500 - 5000 - 10000 - 25000 - 50000 - 75000 - 100000 - 150000 - 250000
Transactional volume for B2C per year: 10000 - 25000 - 50000 - 100000 - 200000 - 500000 - 750000 - 1000000 - 2500000
Color coding B2B in CPO/AOV % for headless backend: red: 1.5 - 100 | amber 1-1.5 | light green 0.5-1 | green 0-0.5
Color coding B2C in CPO/AOV % for headless backend: red: 1 - 100 | amber 0.5-1 | light green 0.2-0.5 | green 0-0.2
Applied discount against standard list price for D365 headless commerce: 55%

Outcome for B2C:

Outcome for B2B:

Conclusions

Although assumptions about affordable CPO/AOV % for headless backend licensing are always arbitrary, I think above matrix at least gives a good overview of percentages of license cost vs Average Order Value (AOV) per order.

What becomes clear from the overviews is that D365 headless commerce engine becomes more and more affordable with either rising volume, rising AOV or both.

In B2C context, a total annual transactional volume of ~500.000 orders (~1370 per day) against a 50-100 AOV already gives you very affordable license cost.

In B2B context, an AOV of 250-1000 already comes with very affordable license cost at low volumes (5000-10000 orders per year).

if your Business does not meet these thresholds, you may try to convince Microsoft to give you a higher discount for headless use, especially when your organization has a hybrid strategy or other strategy which keeps API requests against the CSU lower than average.

About the Author

Get in touch via patrickmouwen.com/contact if you need help in implementing:

D365 Commerce on top of ERP including D365 Store Commerce App and D365 e-commerce
Customizing/Extending D365 Commerce, CSU, Store Commerce App and e-commerce
B2C and B2B Headless Commerce/D365 ERP integrations via Commerce CSU
B2C and B2B Commerce Platform/D365 ERP integrations via Commerce CSU
Customer Service operation based on D365 ERP and D365 CE data utilizing Commerce CSU
Social Commerce/D365 ERP integrations via Commerce CSU
Professional App (React.JS)/D365 ERP integrations utilizing Commerce CSU
POS/ERP integration

Case #2: Can customer account login on your non-Microsoft e-commerce website grant you access to D365 ERP data?

Fri, 22 Nov 2024 00:00:00 GMT

In this series of blog posts I'll share insights, strategies, and proven solutions to address the most common roadblocks and challenges in connecting non-Microsoft e-commerce to D365 ERP.

Today's Case

Q&A

Question raised by the customer: is there a way to directly authenticate our B2B users to access D365 ERP data, just by using their existing login to the e-commerce website?

Short answer: YES! the D365 CSU APIs shipped with D365 ERP support this scenario out-of-the-box. Read on to see how..

External user login to ERP data

e-commerce frontends often have out-of-the-box user account management. In a B2B context, these users are also linked to an organization. Ideally for a seamless user experience, customer log-in would allow your E-SHOP to authenticate directly against D365 ERP data. It would even be better if the ERP data is scoped to the authenticated customer - In other words: that the single customer login in the e-commerce frontend triggers the ERP data to be filtered and narrowed-down to products the customer is allowed to see, with specific pricing reflecting the terms and agreements with the customer.

This is all possible out-of-the-box as shown in the diagram below. The trick is to link the authentication provider's unique identity for a customer to a user and organization in D365 (in B2B world) or customer account in D365 (in B2C world). This setup is then synched to the D365 Commerce CSU as per standard batch jobs, so the CSU is ready to grant customer users access to ERP.

Now, when the user logs into the authentication provider embedded in an E-SHOP page, the E-SHOP will receive a token which contains the unique identity of the customer user which was mapped in ERP and now available in CSU. This allows the CSU to re-validate the token, look up the customer user and organization (B2B) or Customer account (B2C) and grant access to local data in CSU or directly into ERP via a real time connection from CSU. We'll take a deeper dive into these 5 steps below. We'll take the example of a user who logs into a non-Microsoft e-commerce site with Google account. Please note that this will also work with Azure B2C as well as any Open ID CONNECT compatible identity provider such as LinkedIn, Microsoft hotmail, Facebook, Auth0 etc.

Step 1: Setup the Authentication provider [in our case: Google]

Note: prerequisites for below are to have login details of an existing Google account available and have deployed a CSU in case of a sandbox. In case of a DEV-box, the built-in Retail Server will act as CSU.

1A: Setup Google Application to authenticate against D365:

Open https://console.cloud.google.com/ and login with your google account.
Navigate to APIs and Services
Go to the Credentials page
Click "Create Credentials" and select "OAuth 2.0 Client ID"
Configure the consent screen if you haven't done so already
Set the application type to "Web application"
Add https://oauth.pstmn.io/v1/callback to the Authorized redirect URIs
Save and note down the Client ID and Client Secret

1B: retrieve unique identity token representing your customer user

Open a tool to construct and send API requests, such as Postman
Construct a new API request with URL = https://accounts.google.com/o/oauth2/v2/auth
The rest of the URL will be constructed by the Query Params you'll fill in (see picture below)

Do not fire the request from within Postman or other tool you're using, but copy the full URL into a web browser
Login with your Google account
Copy the code after "code=" and before "&scope"

URI decode this code via https://www.urldecoder.org/
Construct another request in Postman to get an ID token with the Authorization code: POST https://oauth2.googleapis.com/token;

This will retrieve an id_token - Copy it as indicated in the picture:

Navigate to jwt.io and paste the id_token
Note down the values for Iss, Aud and Sub - You'll need them in step 2

Explanation:

Step 2: Setup D365 ERP

2A - Setup identity provider

Navigate to Commerce Shared Parameters > Identity providers in D365 ERP
Create a new Issuer record: Type = Open ID Connect - Fill in the value for Iss which you noted down in step 1B - Name can be anything
Create a new relying parties record underneath the new Issuer record**:** paste ClientID from step 1A and set Type = Public and UserType = Customer

2B - Setup for B2C and B2B

Create a new person type of customer account representing the user
Ensure the E-mail address of the customer account matches your Google e-mail address

2C - Setup a B2B customer hierarchy (only required for B2B)

Navigate to the Customer hierarchies form in D365 ERP
Create a new hierarchy
Ensure to configure the customer account from step 2B as a user for the organization and also ensure to list the desired online channel(s) the customer user should have access to

2D - Link Identity provider's identity to a Customer user

Follow the steps in this blog post by Febin Chiramel to create a new record in the RETAILEXTERNALIDTOCUSTOMERMAP table with Excel.
This record will link the Issuer's identity to the customer account created in step 2B. Use the Sub value you noted in step 1B for this.

Step 3: Synch the ERP setup to the CSU

Open the Distribution schedule form in D365 and run jobs 1010 and 1110 to synch the new customer and related identity provider setup to the CSU

Step 4: Login to your Google account on the E-commerce platform

By now, your frontend development partner should have seamlessly made the Google login window available via a web page in your E-commerce platform, so the manual work in step 1B should no longer be necessary.

Step 5: Retrieve D365 ERP Data in real time and scoped to the authenticated customer!

Once the id_token is retrieved by the non-Microsoft e-commerce solution, the token can be used for any subsequent interaction between the frontend and D365 CSU. This is done by putting the token into Authorization header of any API request to the CSU:

The CSU will now re-validate the token by the local identity provider details in the CSU database which you set up in ERP in step 2.
Once the token is validated successfully, D365 will lookup the customer account by the user identity and will contextualize any request against the customer who logged in (!). So for example, a GetActivePrices request will retrieve pricing details (such affiliation pricing) based on the context of the customer who logged in 👍👍👍!

All too complex?

This may seem a bit overwhelming or too complex to some of you. That's why I founded my company 365CONNECT in 2022. Within this company we packaged all our knowledge and experience into a standard Product and Setup offering which makes it very easy to onboard and start using the D365 CSU. See About the Author below.

Licensing and Pricing considerations

A most-heard complaint in the D365 ecosystem in using D365 CSU for connecting non-Microsoft e-commerce to D365 is about the pricing of the CSU. But what if you take into the equasion that your say 800 B2B customer users actually all need access to your ERP data, as this data is directly accessible in the e-commerce frontend? A classical user licensing model would not work here. That's why Microsoft put up a CSU licensing model based on transactional volume and average order volume (AOV). So if your transactional volume and AOV remain constant while your Customer user base grows to 1200 users, you will not pay anything more. I think that's quite fair. We'll further zoom into licensing and pricing in an upcoming blog post.

About the Author

Get in touch via patrickmouwen.com/contact if you need help in implementing:

D365 Commerce on top of ERP including D365 Store Commerce App and D365 e-commerce
Customizing/Extending D365 Commerce, CSU, Store Commerce App and e-commerce
B2C and B2B Headless Commerce/D365 ERP integrations via Commerce CSU
B2C and B2B Commerce Platform/D365 ERP integrations via Commerce CSU
Customer Service operation based on D365 ERP and D365 CE data utilizing Commerce CSU
Social Commerce/D365 ERP integrations via Commerce CSU
Professional App (React.JS)/D365 ERP integrations utilizing Commerce CSU
POS/ERP integration

Case #1: D365 ERP cannot be the backend for your headless e-commerce solution.. Right?

Fri, 15 Nov 2024 00:00:00 GMT

In this series of blog posts I'll share insights, strategies, and proven solutions to address the most common roadblocks and challenges in connecting non-Microsoft e-commerce to D365 ERP.

Today's Case

Q&A

Question raised by the customer: is there really no way to directly connect my headless frontend to D365 ERP? I do not want to synch data since this is costly, incurs latency and leads to issues related to data inconsistency and regulatory compliance.

Short answer: There is a way to do this! The out-of-box APIs shipped with D365 ERP are not suitable to support this like the partner indicates. However, D365 ERP comes with an Add-on called D365 Commerce CSU which is designed to be the desired scalable backend API layer for your headless e-commerce solution.

How D365 ERP can practically be the API backend for your headless e-commerce solution

The Online-Only Retailer will be using D365 ERP in a lot of process domains like Purchasing, Stock Management, Warehousing, Order Management and Finance. All data collected in these areas can be synched out-of-box to the D365 Commerce CSU local database in near-real time (1-5 minutes of recurrence). Some of this data is even available in real time. The dark balloons under backend in the picture below represent this data.

When the Commerce license key, which is included as part of D365 ERP, is enabled, suddenly way more data functionality for your headless e-commerce solution is made availale in ERP (represented by the green balloons in the picture above). The following picture provides a more comprehensive overview of this data and functionality. We'll briefly discuss the additional functionality and data.

1. Additional Backend functions. In the ERP backend, Commerce oriented functions are suddenly enabled in various modules - Examples:

omnichannel tokenized payments: link
Maintaining a navigation hierarchy for your headless frontend: link
AI powered search in your headless frontend: link
Item warranty: link

2. Making this functionality and data available. All of this data and functionality can be synched to the Commerce CSU as per an out-of-box synch framework. Data in different areas can be synched with different frequency, depending on required actuality. In many cases, there is also a real time alternative available in case your frontend requires real time data to be presented to the customer. Vice versa, new Orders coming in from the headless frontend are synched from CSU to ERP out-of-box as well.

3. Commerce CSU APIs. Where D365 ERP has an API layer which is mostly Data oriented (to read, write, delete and update data), the Commerce CSU API is packed with Business logic. This means that the CSU APIs can calculate prices, recognizes customer, channel and legal entity context and even integrates out-of-box with various AI functions like Azure AI Search for products and Product Recommendation service.

In upcoming blog posts, I will explain how to benefit from the APIs in a variety of domains such as Inventory management, Pricing Management and Product Management.

Advanced Scenarios

Different Businesses require different setups. And headless e-commerce setups may differ by nature as well. Below we'll take it one level deeper looking at 3 advanced headless scenarios and how D365 Commerce CSU can support these.

Scenario #1: our Online-only Retailer expands to Asia and Americas

When expanding their Business to the Americas, the Retailers wants to keep latency to a minimum. To accommodate this, multiple CSUs can be deployed. For example, one in Europe and one somewhere in the American region. D365 ERP still allows all data to be synched to both CSUs or the company can make a selection of the data per CSU.

Scenario #2: our Online-only Retailer decides to implement a Progressive Web app

A Progressive Web App (PWA) is a type of web application designed to provide a fast, reliable, and engaging user experience similar to that of native mobile apps, but accessible directly from the browser. To minimize latency, web traffic is to be routed from client to backend API layer directly. But sensitive data should be routed server to server. By utilizing API roles which are embedded in D365 CSU out-of-box, both scenarios can be mitigated: the Customer role is suitable for authenticating an individual shopper. The Application role is suitable for securing server to server communication. We’ll explore both roles in upcoming blog posts in this series.

Scenario #3: our Online-only Retailer requires API performance under 100 ms

If CSU API performance is not sufficient for some frontend journey components, the customer may choose to implement an in-memory cache. This can either cache CSU requests for quick re-use in subsequent frontend interaction (option 1 in below picture) or can be sourced from D365 ERP by custom batch jobs (option 2 in below picture). This can lead to a hybrid setup where most headless e-commerce components are directly sourced via the Commerce CSU and others indirectly via the in-memory cache.

About the Author

Get in touch via patrickmouwen.com/contact if you need help in implementing:

D365 Commerce on top of ERP including D365 Store Commerce App and D365 e-commerce
Customizing/Extending D365 Commerce, CSU, Store Commerce App and e-commerce
B2C and B2B Headless Commerce/D365 ERP integrations via Commerce CSU
B2C and B2B Commerce Platform/D365 ERP integrations via Commerce CSU
Customer Service operation based on D365 ERP and D365 CE data utilizing Commerce CSU
Social Commerce/D365 ERP integrations via Commerce CSU
Professional App (React.JS)/D365 ERP integrations utilizing Commerce CSU
POS/ERP integration

Commerce CSU V9.51/10.0.41 - November 2024

Wed, 13 Nov 2024 00:00:00 GMT

For anyone new in this field: Commerce CSU is shipped as part of D365 ERP and can seamlessly integrate non-Microsoft e-commerce with D365 ERP:

I have been using Commerce CSU for e-commerce integrations since its early days around 2015. My mission is to empower you and your organization to use Commerce CSU effectively for your B2C and B2B e-commerce integrations. Therefore, I will publish these unofficial CSU Release Notes with any upcoming new Release.

A-Release Note Details

These Release notes are applicable for the following CSU and D365 ERP version:

B-Downloads for your convenience

Latest V9.51 CSU full interface specification in OpenAPI/Swagger format: Link

A snippet of the more than 350 APIs in Commerce CSU

Lestest V9.51 Overview of required Commerce Roles per API: Link

A snippet of the role identification for the more than 350 APIs

C-For the Techies: API and Model Changes compared to the previous version 9.50/V10.0.40

C.1 New APIs

None

C.2 Removed APIs

None

C.3 Data Model Changes

Comments:

Change #3 - DisplayNonRetailOrdersInEcommerceOrderHistory: is a new parameter in D365 ERP backend > Commerce Parameters - It displays or hides Orders which have been created with the Supply Chain Pricing engine as opposed to the Commerce Pricing engine:

D-About the Author

Get in touch via patrickmouwen.com/contact if you need help in implementing:

D365 Commerce on top of ERP including D365 Store Commerce App and D365 e-commerce
Customizing/Extending D365 Commerce, CSU, Store Commerce App and e-commerce
B2C and B2B Headless Commerce/D365 ERP integrations via Commerce CSU
B2C and B2B Commerce Platform/D365 ERP integrations via Commerce CSU
Customer Service operation based on D365 ERP and D365 CE data utilizing Commerce CSU
Social Commerce/D365 ERP integrations via Commerce CSU
Professional App (React.JS)/D365 ERP integrations utilizing Commerce CSU
POS/ERP integration

Disclaimer

For authoritative product information, consult Microsoft's official documentation at learn.microsoft.com.

What is new: D365 Commerce CSU

Mon, 22 Jul 2024 00:00:00 GMT

Introduction

There are no official Microsoft Release Notes on the latest D365 Commerce (Retail) Cloud Scale Unit (CSU) APIs. Since more and more companies are starting to use these APIs for integrating external e-commerce websites with D365 ERP, I think the community may appreciate some 'unofficial' Commerce CSU "WHAT'S NEW" 😎.

Besides that, the 'official' list of APIs is not really kept up-to-date here on Microsoft learn, so below you'll find links to the latest API documentation, an up-to-date list of all APIs and what role they're designed for (Customer = website, Employee/Device = store, Application = server-to-server).

Release notes | Latest version

The latest Commerce CSU version which is used for the Release notes here is:

The version is compared with 9.49 (10.0.39).

Removed APIs

/Commerce/GetOrdersByStore

/Commerce/GetPickUpOrdersByStore

New APIs

Note on Roles vs Use:

Anonymous/Customer/BusinessPartner = e-commerce
Employee/Device = In-store
Application = Server-to-Server

As per this list, it becomes clear that Microsoft is investing to expose Copilot/ERP backend generated summary information through the CSU. Unfortunately, this is currently only designed for use in-store (role = employee), but it should be possible to extend this.

Data Model changes

The list above demonstrates investment in the following areas:

Self checkout POS (in-store): more detail
Copilot Summaries: more detail
New search criteria for stock availability

Relevant links

All Commerce APIs and roles (V9.50) in Excel: link

Full Commerce API documentation (V9.50): link

About the Author

Get in touch via patrickmouwen.com/contact if you need help in implementing:

D365 Commerce on top of ERP including D365 Store Commerce App and D365 e-commerce
Customizing/Extending D365 Commerce, CSU, Store Commerce App and e-commerce
B2C and B2B Headless Commerce/D365 ERP integrations via Commerce CSU
B2C and B2B Commerce Platform/D365 ERP integrations via Commerce CSU
Customer Service operation based on D365 ERP and D365 CE data utilizing Commerce CSU
Social Commerce/D365 ERP integrations via Commerce CSU
Professional App (React.JS)/D365 ERP integrations utilizing Commerce CSU
POS/ERP integration

Contact me on LinkedIn for more details: https://www.linkedin.com/in/patrickmouwen/

Disclaimer

For authoritative product information, consult Microsoft's official documentation at learn.microsoft.com.

Reconciling Adyen vs D365 Payment Transactions

Thu, 12 Oct 2023 00:00:00 GMT

D365 F&O/Commerce offers an out-of-box Adyen payment connector for use In-Store (“card present/card not present”), in Call centers or Online (“card not present”). When payments are physically captured through this connector, both ‘sides’ (Adyen and D365) capture ‘evidence’ for the physical transaction. In many situations it can be hepful to reconcile this ‘evidence’. For example, when a third-party e-commerce solution and D365 Store Commerce App both go through an Adyen Merchant account, when payments have been captured offline (so by manually inputting an amount on the Adyen payment device) or simply to proof that all Payments and related Sales have made it into D365 F&O HQ.

In this blog post I’ll describe how you can reconcile the Payment transactions between Adyen and D365 F&O.

Read the full story here in my Linkedin article

D365 Commerce + IOM = SUPERPOWER!

Fri, 29 Sep 2023 00:00:00 GMT

Microsoft Intelligent Order Management (IOM) is a powerful solution for effectively orchestrating Order Fulfillment scenarios.

D365 Commerce is D365 Finance & Operations plus the Headless commerce engine with more than 400+ Commerce APIs and a local database which is installed on a separate Retail Cloud Scale Unit.

The combination of IOM + D365 Commerce offers next level capabilities for any Retail Enterprise 🚀.

In below video I’ll demonstrate an integrated scenario in which Orders from an E-commerce platform are orchestrated in IOM and fulfilled via D365 Commerce.

Why I am a fan of a Single Company setup for Retailers in D365 F&O

Fri, 29 Sep 2023 00:00:00 GMT

Implementing D365 F&O requires some specific Design considerations when it comes to Retail organizations. A Retail company is not simply organized by Legal entities which transact with each other and 3rd parties. A Retailer has one or more brands to protect. Consumers expect the same brand experience anywhere anytime. This experience cannot have interference from legal entity boundaries or the way an ERP system is configured.

Too often, I see D365 F&O Solution Architects and Consultants automatically translate a Retailer's subsidiaries to companies in D365 F&O. Even if this does not align with the way the Retailer is organized (central backoffices) and/or their logistics model is setup (hub and spoke). But first and foremost, it can shatter Supply Chain and Order fulfillment into pieces and significantly raise operational complexity and overhead with detrimental impact on the Customer Journey..

Is there an alternative???

The Retailer's dilemma: Vertical walls vs Horizontal freedom

Retail organisations reduce financial risk by slicing their operation into legal entities organized by brand, specific activities and/or jurisdiction. In doing so, If any of the activities are underperforming potentially causing a bankruptcy, it can never bring down the complete enterprise.

At the same time, Retail companies require boundless operation to effectively manage their supply chain from Purchasing to Retail and to provide a consistent high-quality experience to their consumers, regardless of brand, country or specific store.

These 2 powers of ‘vertical structure’ vs ‘horizontal freedom’ brings a field of tension in typical cross-border Retail Businesses which is most prevalent in Retail Businesses with brick-and-mortar (stores) and/or service activities.

When defining the Financial setup in D365 F&O for these Retail companies, most Dynamics partners translate the company structure 1:1 based on classical in-operation intercompany postings (1# in the picture below) and then try to make the best out of the ‘horizontal freedom’ dimension (#2 in the picture below).

It is perfectly understandable to choose this path since any Solution must be legally and fiscally compliant. However, this comes with a price as it shatters the Supply chain and Order Fulfillment into pieces. Stitching these together to enable effective management of the Supply chain and seamless Customer journey is complex and costly.

Shattering Order Fulfillment into pieces?

Retail companies are often legally organised along the following lines:

Holding company: often stock holding company, so responsible for buying and stock (re)distribution - Sometimes e-commerce activities are also operated from this company
Brand holding: parent company for all brand specific activity
Selling company per brand/country: this company holds all sales activities like store sales, online sales
Service company per brand/country: this company holds all field service activities like on-site advisory, installations, repairs etc.

Often this legal entity structure (or similar structure) is translated into a 'classic' Intercompany structure in D365 F&O. In other words: each Legal Entity is reflected by a company in D365 F&O and every transaction between these companies is reflected by an additional Intercompany Sales Order/Purchase order pair (=IC PO/SO). Relatively straightforward Business processes can then suddenly become utterly complex to manage.

Example Retail Business Process.

In this example, a consumer places an in-store order for a Make-To-Order item (such as a bespoke closet or suit). The item is to be purchased from a supplier and then shipped to the consumer, in this case by cross docking via a main warehouse.

D365 F&O (including Commerce) is not too strong when it comes to managing more complex Fulfillment journeys. For these cases I would definitely recommend D365 Commerce/F&O to Team up with D365 Intelligent Order Management (=OMS). See for example my video on the combination of IOM and D365 Commerce. With a "classic" intercompany setup, the Fulfillment journey is even more complicated due to the additional ICO SO/PO pairs.

In this example, not 1 SO (Sales Order) with related PO (Purchase order) and TO (Transfer Order) is to be managed, but 3 SOs, 3 POs and 1 TO. And there are more complexities like the fact that stock is only available in Legal Entity 2 (the main holding) where we'd like to reserve stock from the main Sales order in Legal Entity 1. Even if D365 Intelligent Order Management (IOM) would be used here, IOM would have to manage and monitor multiple 'overhead' intercompany documents. This compromises "Keeping the Promise" which raises complexity in OMS implementation and OMS operation.

Beyond all this impact on the Fulfillment journey, there are more considerations which urge to look for an alternative set up. For example:

Rapid D365 F&O DB growth by ICO transactions. Customers now pay for this DB footprint!
Impact on Customer journey: Only Legal Entity 1 in the example is aware of the Consumer - The consumer does not exist and is not known in Legal Entity 2 or 3.
Customisations for IC processes. Some fundamental requirements for effective Order Fulfillment and Order Management are not compatible in a D365 F&O intercompany context. These will require (heavy) customisations in D365 F&O increasing implementation risk and Project implementation cost.
Organizational alignment: Retail organizations often have a central Backoffice per group of countries (=Region). These backoffices have to manage pieces of the Fulfillment chain across multiple D365F&O companies which is not ideal.
Margin: in an Intercompany Setup, margin components (like landed cost, shipping cost and fulfillment cost in general) are not sitting in 1 D365 F&O company. Of course, this can be stitched together in (for example) the BI domain, but some stakeholders prefer to see all cost components in their financial records.
You can probably come up with more downsides yourself!

Finding an Alternative Setup in D365 F&O

First let's set a goal.. In the ideal world, we'd keep things as SIMPLE as possible. For a Retailer this would mean an all-in-one company setup in D365 F&O which hosts their Full Operation, from buying to order fulfillment. It would also mean to have no in-operation intercompany transactions, but post-operation intercompany postings, aggregated as per desired level (i.e. transactions condensed daily, monthly or quarterly).

But how is this possible while still addressing all specific Financial Requirements the enterprise may have AND being fully compliant with legal and fiscal regulations...?

Lets keep some typical Financial, Legal and Fiscal requirements in mind when trying to find an alternative for the "classic" intercompany setup:

Typical Financial, Legal and Fiscal requirements.

The D365 F&O Single "OPCO" (Operating Company)

My best shot to realise the Single company goal and being compliant with Financial, Legal and Financial requirements is the so called "OPCO" (Operating Company) concept.

In this company all 'normal' legal entities are reflected by D365 F&O companies, but they do not have direct intercompany relationships. Instead, the full Retail Operation is happening in the OPCO, a company in D365 F&O which does not represent a real-life Legal Entity.

Characteristics of the OPCO

Financial Dimension Values. All transactions in the OPCO are stamped with Financial dimension values which at least represent Brand, Cost center and Legal Entity. Non-D365 Commerce consultants may be surprised by the D365 Commerce out-of-box capability to allow inheritance of financial dimension values from Retail channel to Sales order/Sales order line. This auto-stamps the right Brand and Legal Entity for any Sale. Knowing these details on any Transaction will also allow to print a Brand and even Channel specific invoice and receipt if required. Since this information baseline provides full traceability and accountability, there is a solid fiscal and legal foundation for invoicing the consumer from the OPCO.
Statutory Reporting. Intrastat, VAT declaration, EU Sales list and other statutory reporting are directly created from the OPCO. The 2022 introduced functionality to report for multiple VAT registrations from 1 company is utilized for this - This is now supported for many European countries. See Microsoft Learning.
Intercompany transactions. IC Transactions are posted POST-Operation from OPCO to the actual Legal Entities. This is done by setting up Intercompany Accounting and Ledger allocations which automatically offset post OPCO financial transactions to the actual Legal Entity by main account/financial dimension rules. So for example, financial postings on OPCO revenue accounts can be broken out to various postings to the ledgers of the actual legal entity, so a P&L can be viewed from that Legal Entity.
OPCO = Stock holding company. Optionally the OPCO is the Stock holding company itself, so all companies in D365 F&O align with real-life Legal Entities.
Multi-currency. The OPCO can cover jurisdictions with different primary currencies. D365 F&O is dual-currency system since a few years now, so this should also not be a deal breaker if the number of primary currencies does not exceed 2.

Challenges

The Single Company/"OPCO" concept is often quickly wiped off the table by hardened D365 F&O Consultants and Solution Architects. This is often because they have done their intercompany transaction routine for so many years that they do not even consider other options. In other cases, it's because they don't know how to overcome the following 2 most-heard obstacles:

Localization. The DataAreaId (=company identifier) in D365 F&O is not only used in tables, to slice Data into Legal Entity specific buckets or shared buckets. The Legal Entity is also used to trigger locale (=jurisdiction) specific code. In most cases, this code is triggered for groups of countries which have similar legal context, such as the following regions: BeNeLux, DACH, Scandinavia, Ireland/GB and Eastern Europe. My rule of thumb is to have 1 OPCO for each of these regions. This also aligns with the Retailer's organization and Logistics model in most cases: when a Retailer operates in BeNeLux (Belgium-Netherlands-Luxemburg) and expands brick-and-mortar (store operation) into DACH (Germany-Austria-Switzerland), often a new local (DACH) organization and logistics hub is set up for that region:

Subledger traceability. Especially financial auditors and controllers have the requirement to be able to click from the Legal Entity's General ledger to the OPCO's General ledger and Subledger. If the OPCO>Legal Entity intercompany accounting postings are condensed, this cannot be arranged on a 1:1 transactional basis. In that case, click-through capability is to be arranged by (minor) customization, for example by clicking from a single non-OPCO General ledger entry into multiple OPCO subledger transaction.

Final words

Every Solution Design choice should always be made in the specific context of an Enterprise. On top of that, a Project stakeholder will always be able to find a weak point in any Design Decision. But in our profession as an IT Architect it should not be the aim to pick the best option in a specific domain. The aim is to pick the best option Cross-Domain, where the overall score/value of the Solution as a whole is the highest possible.

I hope this post inspired you to look beyond 'proven Design paths' which may not represent the best Overall score.

In case of Retail organizations, the decision for a Single company setup or small group of OPCOs instead of 20+ or even 100+ companies in D365 F&O is more than fundamental: it can be the difference between a highly successful Transformation Program and a Program which technically succeeds but is considered a pain in the a** by the Retail Operation!

D365 F&O/Commerce interfacing via Azure API Management: My Best Practices

Mon, 28 Feb 2022 00:00:00 GMT

I am seeing more and more customers and partners who actively promote the use of Azure API Management for D365 F&O backend and/or D365 Commerce Scale Unit OData interfacing. See for example, this blog post by Adrià Ariste Santacreu. However, there’s still a significant number of customers and partners who shrug their shoulders while thinking: “Why would we use it when we can interface ‘directly’?”. In this blog post I’ll address this question by sharing my best personal best practices in using Azure API Management for D365 F&O backend and/or D365 Commerce Scale Unit OData interfacing. But before sharing the best practices, we first have to explode the myth that interfacing via Azure API Management (APIM) would not be DIRECT.. 😉.

1 – The role of Azure API Management in D365 F&O interfacing

Imagine a car which is driving from A to B every day. In order to keep it on the road it’s needs to be fueled, maintained and cleaned. When this handling is done, the car is grounded and cannot move from A to B. How cool would it be if the car could drive through an advanced car wash where the fueling, maintenance and cleaning could be done while keeping it’s pace and direction on its way to B. This is actually what Azure API Management (APIM) is about: consider APIM as a gateway (a tunnel – see Figure 1 below) which interface requests are routed through on their way to a backend system.

Figure 1: Client to D365 F&O traffic going through the APIM tunnel
Source: Microsoft Docs (edited)

While passing through the APIM gateway, ‘Value Added Services’ are applied to the ‘vehicle’ (=the request). These ‘VAS’ hardly slow down the vehicle, often only by about 25-100ms. These ‘VAS’ are the policies which are applied to either the incoming request, request to the backend (the 2nd part of the tunnel) and/or the equivalent responses which are routed through the same tunnel. So is the call from the client to D365 F&O still DIRECT? I’d say YES.. its only running through a tunnel of ‘VAS’… 😉.

2 – Why using APIM with D365 F&O?

For many reasons.. Let’s start by taking a few basic principles of interfacing: first Authentication. Clients which interact with D365 F&O can be very different in nature. Often it’s a non-Microsoft system backed by a partner who is not used to Microsoft platform standards. Take for example an e-commerce system and partner. It sounds strange to many but I’ve met a significant number of partners who are only used to interface with API Keys and Certificates and had never implemented oAuth 2.0 authentication before. D365 F&O would force them into oAuth 2.0 authentication which may take longer than usual for them to adopt. With APIM, APIM can take care of the authentication to D365 F&O, independent from the Client to APIM authentication.

Another example is Error handling.. Imagine if 20 different systems interact with D365 F&O where each of them has their own standard in error handling. We want STANDARDISATION and CENTRALISATION of error handling. We don’t want system A sending us e-mails when interaction with D365 F&O runs into an issue, system B offering an error on a management portal and system C not even surfacing an error (we have to find out in D365 F&O that we’re missing data..). With APIM, we can configure APIM to report errors to Azure App Insights which can source as a baseline for centralised error management.

My last example is API documentation. As we all know, neither the D365 F&O backend OData end points nor the D365 Commerce Retail Server OData end points are OpenAPI or Swagger documented. When we configure APIs with their Operations in APIM, we can include all properties of the APIs including sample requests and all possible status code/response body pairs:

Figure 2: Azure API Management: define possible Status codes and Responses

These properties are automatically published to a Developer portal with all information about our APIs. We can empower IT Partners and other stakeholders to become self-sufficient in consuming our APIs by allowing them access to the portal information for some or all APIs we’ve technically documented in APIM:

In the next section I’ll share my personal best practices. I’ll already tell you this: you’ll be able to distill a lot of benefits for use in D365 F&O/D365 Commerce Scale Unit interface scenarios from them.

3 – My best practices in using APIM for D365 F&O and D365 Commerce Scale Unit integrations

There are many things to share in regards to APIM specific best practices. Below categorisation is narrowed down to my personal best practices in using APIM for D365 F&O and Commerce OData endpoint integrations specifically:

Abstraction
- Backend system
- Environment
- Template parameters
Security
- Template parameters
- Strip query parameters
- Mask backend URL
Authentication
- Decouple client to APIM vs APIM to backend authentication
- Auto rotate, cache and re-use oAuth2.0 bearer tokens
Performance
- Retry policy
- Caching
Flexibility
- Re-pointing clients to another D365 environment in under 10 seconds
- Using APIM end points as opposed to the D365 F&O connector in Power Automate or Power Apps
- API Versioning
API Documentation
- Developer portal
- Documentation in Power Automate – Logic Apps – Power Apps

3.1 – Abstraction

Azure API Management acts as a facade, a shell, for your system end points – If implemented well, this shell actually turns your technical end points into organizational Services which can easily be consumed by internal users, internal applications and partner applications. This is achieved as follows:

External clients can become system agnostic:
- Clients no longer have to bother which Application a particular function originates from. Instead, the clients can directly start to consume the functions. In other words: it allows a Service Oriented Architecture (SOA).
- In the case of D365 F&O and Commerce, APIs can be exposed as Services which can easily be consumed by any App, Power Automate Flow or Backend system. These Services can be organized as per organizational standards similar to defining a custom namespace in a .NET library:

External clients can become system agnostic (continued)
- When organising your end points as Services, the backend system becomes irrelevant. You could replace the endpoint by an updated endpoint or endpoint from another system and your client wouldn’t even notice it. So from client perspective a D365 specific GET {D365 environment}/data/CustomerPostalAddresses call is replaced by a generic GET {customername}.azure-api.net/customers/getcustomeraddresses. For D365 F&O, this can be achieved by implementing below redirect policy on the particular API Operation in APIM:

Policy: set backend URL suffix

API > Specific Operation > Inbound -> rewrite-uri

<inbound>
        <base />
        <rewrite-uri template="/CustomerPostalAddresses?$filter=CustomerAccountNumber eq &apos;{CustomerAccount}&apos; and dataAreaId eq &apos;{Company}&apos;&amp;cross-company=true" copy-unmatched-params="false" />
</inbound>

External clients can become environment agnostic: clients no longer have to bother where a particular function comes from. Instead, the clients can directly start to consume the functions. Client requests can be re-directed to another environment by an APIM config change in seconds.
- This can be achieved by configuring a Named Value in APIM in conjunction with a Policy – In this case I put the policy on the API’s All Operations so it applies for all Operations for a particular API (here: D365 F&O entity Customers):

Named Value: D365 F&O backend URL or D365 Retail Server URL

APIM > APIs > Named Values

Policy: set backend base URL

API > All operations > Inbound -> set-backend-service base-url

<policies>
<inbound>
	<base/>
	<set-backend-service base-url="{{D365OPSEnvironment}}/data"/>
</inbound>

Requests and request parameters can be simplified. For example, when searching for a customer address in D365 F&O, an OData call can become very complex when searching by multiple columns. In APIM, the most common searches can be standardised by defining multiple API operations which offer different sets of criteria. For example:
- API Operation 1: getaddresses -> Allows searching by street, city and zip code
- API Operation 2: getcustomeraddresses -> Allows searching by customer account
- Each specific Operation offers specific search options (APIM: Template parameters).
- The formatting of these search options can be simplified for the user, for example:
  - D365 F&O request:
    /CustomerPostalAddresses?$filter=CustomerAccountNumber%20eq%20’CUST0155148’%20and%20AddressLocationRoles%20eq%20’*Delivery*’&cross-company=true
  - Same request to D365 F&O via APIM:
    /customers/getcustomeraddresses**?customeraccount=CUST0155148&company=CMP**
- This can be achieved by defining the respective Template Parameters on the API Operation in conjunction with the Rewrite URL policy as presented under point 2 above:
  - Having these template parameters defined is also handy when using the Test feature in APIM

3.2 – Security

Template Parameters + Strip off any additional parameters: the use of Template Parameters as shown under 3.1 point 4 is a strong security measure when combined with a policy to strip off any additional query parameters – For example, if a GET type of API Operation is defined to only allow searching for Customer addresses by Customer account, APIM will not forward criteria which would not comply with this even if the respective backend system like D365 F&O supports it (e.g. Customer name). In doing so, only clients which are aware of actual Account number can fire a search request. A hacker firing calls by potential Customer names will not even be forwarded to the backend as such.

Policy: Strip off Query parameters

API operation > Inbound > Rewrite-URI Policy -> copy-unmatched-parms=false

<policies>
    <inbound>
        <base />
        <rewrite-uri template="/CustomerPostalAddresses?$filter=CustomerAccountNumber eq &apos;{CustomerAccount}&apos; and dataAreaId eq &apos;{Company}&apos;&amp;cross-company=true" copy-unmatched-params="false" />
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Mask backend URL: when a call is fired to D365 F&O and Commerce, both return the URL of D365 F&O and the Retail server in the response. For security reasons, we don’t want the base URL to be exposed to the client as this might enable a malicious client to fire a request to the backend directly. With the following policy any base URL in the response is replaced by the APIM base URL:

Policy: mask backend URL

API > All operations > Outbound -> redirect-content-urls

<outbound>
	<base />
	<redirect-content-urls />
</outbound>

Now when D365 F&O or Retail Server return a response, the respective backend URL is replaced by the APIM base URL:

{
	"@odata.context": "https://myAPIM.azure-api.net/Customers/$metadata#CustomerPostalAddresses",
	"value": [
		{
			"@odata.etag": "W/\"JzEsNTYzNzgwNjA3Nic=\"",
			"dataAreaId": "COMP",
			"CustomerAccountNumber": "CUST0155148",

3.3 – Authentication

APIM can take care of the authentication to the backend system. This has a couple of benefits:

Security: sensitive security information is not distributed outside Azure:
- Decoupling Client to APIM and APIM to Backend authentication saves you from having to distribute sensitive information like a Client ID/Passphrase to a client application.
- It is even possible to setup oAuth 2.0 (Client ID/Passphrase) to APIM and separate oAuth 2.0 (Client ID/Passphrase) from APIM to D365 F&O or Commerce Cloud Scale Unit – In this way the access to D365 resources via APIM can be authenticated differently per client. Vice versa, a client can have 1 identity/login for all resources (D365 and non-D365) in your tenant. In other words: a client could login to all relevant resources on your tenant by a single APP ID/Passphrase even though one or more backend systems would not support oAuth 2.0 (!).
- In case a client connects to APIM via API-Key, the API-Key in the request can be stripped off to not come through to the backend system. This ensures the API-Key cannot be intercepted beyond APIM and then be re-used in a new client request:

Policy: strip API-Key header off request to backend

API > All operations > Inbound -> set-header -> “delete”

    <inbound>
        <base />
        <set-header name="API-Key" exists-action="delete" />
    </inbound>

Background rotation, caching and rotation of oAuth2.0 bearer token:
- When APIM is used to authenticate to the D365 backend or D365 Commerce Cloud Scale Unit independent from client authentication, the token for accessing the backend can be cached and auto rotated for faster response times.
- This can be achieved by a combination of a parameter for Token life time and inbound policy
- Note: the bearer token is always specific for a resource – Hence, my best practice is to save the token into cache by Environment name – In doing so, the token is automatically refreshed when APIM is redirected to another D365 environment
- Note 2: I often save the Client ID and secret in Azure Key vault and use that reference in the respective Named value (see column Key vault name below)

Named Value: Client ID – Secret – Environment – Environment name -Token life time

APIM > APIs > Named Values -> Token life time setting 20 minutes = 1200 seconds

Policy: fetch oAuth2.0 token, cache and re-use

API > All operations > Inbound ->
send-request
set-header
cache-store-value
cache-lookup-value

    <inbound>
        <base />
        
        <cache-lookup-value key="{{D365OpsEnvironmentName}}-token-{{ClientId}}" variable-name="bearerToken" />
        <choose>
            
            <when condition="@(!context.Variables.ContainsKey("bearerToken"))">
                <send-request mode="new" response-variable-name="oauthResponse" timeout="20" ignore-error="false">
                    <set-url>{{AuthorizationServer}}</set-url>
                    <set-method>POST</set-method>
                    <set-header name="Content-Type" exists-action="override">
                        <value>application/x-www-form-urlencoded</value>
                    </set-header>
                    <set-body>@("grant_type=client_credentials&client_id={{ClientId}}&client_secret={{ClientSecret}}&resource={{D365OPSEnvironment}}")</set-body>
                </send-request>
                
                <set-variable name="accessToken" value="@((string)((IResponse)context.Variables["oauthResponse"]).Body.As<JObject>()["access_token"])" />
                <cache-store-value key="{{D365OpsEnvironmentName}}-token-{{ClientId}}" value="@((string)context.Variables["accessToken"])" duration="{{TokenLifeTimeSeconds}}" />
                <set-header name="Authorization" exists-action="override">
                    <value>@("bearer " + (string)context.Variables["accessToken"])</value>
                </set-header>
            </when>
            
            <otherwise>
                <set-header name="Authorization" exists-action="override">
                    <value>@("bearer " + (string)context.Variables["bearerToken"])</value>
                </set-header>
            </otherwise>
        </choose>
    </inbound>

3.4 – Performance

Retry policy:
- I actually do not prefer to let APIM handle retries to backend systems:
  - Requests which can take a long time (say > 10 seconds) may even take longer when initial attempts fail. This can push the client into a time out window.
  - Most 3rd parties have their own retry policies incorporated into their API framework. This is for the client to stay in full control over any call.
- If your client application framework does not support re-try policies or you’d like to utilise the advanced options APIM offers for retries, you can enable a retry policy in APIM as per below example:
  - Note: after V10.0.13 D365 F&O has throttling enabled for OData requests
  - As part of this framework, D365 F&O will return status code 429 “Too many requests” in case of request overkill. So we can put a retry condition on that status code to diversify the re-try settings per status code (!):

Policy: retry requests to backend systems

API > All operations > Outbound -> retry
Note: I would recommend to put the count and interval variables in Named values, so an update to the named value applies to all API operations at once

 <outbound>
        <retry condition="@(context.Response.StatusCode == 429)" count="3" interval="10" max-interval="" delta="" first-fast-retry="false" />
 </outbound>

Caching: responses can be cached
- If a client application is firing the same request within a certain timeframe, APIM can return a cached response as opposed to forwarding the call to the backend system again
- I’ve used this pattern in an e-commerce to D365 Commerce integration scenario: customers tend to click back and forth between pages which can lead to re-loading of the same data multiple times. Returning cached responses was very useful here to reduce load times and number of calls to the D365 Cloud Scale Unit.
- A cached response takes about 45-50ms to be returned to the client, so it leads to improved performance in repetitive scenarios
- Here’s how to implement it:

Named Value: time window to cache a response for

APIM > APIs > Named Values -> Caching duration

Policy: cache a response and return it from cache

API > All operations > Outbound -> cache-store
API > All operations > Inbound -> cache-lookup-value

    <inbound>
        <base />
        <cache-lookup-value key="{{D365OpsEnvironmentName}}-token-{{ClientId}}" variable-name="bearerToken" />
    </inbound>
    <outbound>
        <base />
        <cache-store duration="{{CachingDurationInSeconds}}" />
    </outbound>

3.5 – Flexibility

Re-pointing to another environment: since APIM allows client applications to consume Services backend system agnostic, client apps can be linked up with a different backend system without downtime.
- When implemented as descibed under #1 Abstraction above, re-pointing is just a matter of changing the value for parameters D365OPSEnvironment and D365OPSEnvironmentName. Since APIM won’t find a bearer token for that Environment name, it will retrieve a new one automatically.
- With this, it’s easy to re-point a TEST e-commerce website from a D365 F&O DEV box to a sandbox for example
Using APIM end points in Power Automate, Logic App and Power Apps as opposed to the D365 F&O connector:
- The D365 F&O connector for Power Automate/Logic Apps simplifies integration scenarios for non-techies
- However, using the connector is not too simple due the requirement to come up with the right entity key pairs in update/insert/delete scenarios
- In addition, using the connector requires a User based data connection where APP based authentication might be preferred
- Using an APIM endpoint as opposed to the D365 F&O connector saves a data connection in the Flow or Logic App orchestration – This makes it more ‘lightweight’ and easier to export/migrate, for example to another tenant
- See below how to create a HTTP request to a D365 F&O OData endpoint to fetch all the address information for a customer via APIM as per the setup described in this blog post – For authentication to APIM, API Key is defined – APIM authenticates to D365 F&O by bearer token as described under #4 – Authentication:

Version your APIs: when in Production it’s important to have the ability to quickly jump over from one API version to another for patching purposes while staying risk-free. To achieve this you can enable versioning on any API:
- Enable versioning by choosing the option Add version on any of your APIs (see picture below)
- Now, your client application can easily jump over to a new/patched version of your API by a simple URL change: e.g. switching from {APIM URL}/customers/getcustomeraddresses to {APIM URL}/customers/V2/getcustomeraddresses

6 – API Documentation

Developer portal: Both D365 F&O OData and D365 Commerce Cloud Scale Unit OData interface frameworks lack OpenAPI/Swagger documentation:
- To fill this gap, Microsoft ship proxy clients for D365 Commerce OData and OData V4 Client Code Generator tool for Visual Studio which can be used to generate a Client proxy for the D365 F&O backend OData service
- However, these proxies only allow Programmatic Integration and are more generic tools which may lack some detail
- When a D365 F&O or D365 Commerce API is described in APIM in terms of template parameters, header parameters, request body, potential status codes etc., this information is automatically published to the APIM Developer portal
- The Developer portal can be fully customised and customer branded, similar to SharePoint sites
- Internal and external users can be authenticated (either via e-mail/password or AAD user login) to access all or a subset of APIs so they become sulf-sufficient in consuming them:

Azure API Management Developer portal: available APIs after login

APIM > Portal overview > Developer portal

Azure API Management Developer portal: API details

APIM > Portal overview > Developer portal

Azure API Management Developer portal: Export to WADL, OpenAPI v2 or v3

APIM > Portal overview > Developer portal

Documentation in Power Automate – Logic Apps – Power Apps: APIM APIs can be exported as Power Connector so the API documentation is also available in Power Automate, Logic Apps and Power Apps:

APIM: export as Power connector

API > Create Power Connector

Power Automate – Logic App – Power Apps

List of APIs pulled from APIM definitions

Power Automate – Logic Apps – Power Apps

Custom APIM based connector with PIM documentation
Note: the connector only requires API-Key authentication to APIM as per the respective definition in APIM

4 – Final words

I hope this blog post inspired you to explore the use of Azure API Management for D365 F&O and D365 Commerce interfacing and more. I think APIM can be useful in many scenarios. Fortunately, the cost are relatively low. A customer can already deploy a Production environment for a monthly cost of just over 150 dollar per month.

Warm regards,
Patrick Mouwen
03.09.2022

How to save your D365 F&O customer some money

Wed, 23 Feb 2022 00:00:00 GMT

In most projects, customers and partners are deploying Build and DEV boxes from LCS\Cloud hosted environments. These utilise multiple Azure resources such as storage accounts, virtual network and virtual machines. In many cases, the LCS environment is working against only 1 Azure subscription which is eventually used to host the PROD environment as well. If this is the case, you can potentially save your customer quite some money. In this blog post, we’ll see how much and how!

1 – How much money can your customer save on Azure consumption?

Open Chrome and open the Azure pricing calculator: https://azure.microsoft.com/en-us/pricing/calculator. Now select Virtual Machine and pick 1 virtual machines with the following specs (which is default in LCS):

Instance: D13 – 8vCPU – 56GB- 400 GB
Tier: Standard HDD
Disk Size: 3xS15: 256GiB
Virtual Machines: 1

Pick the following details for calculating the monthly cost against standard Azure pricing:

Number of hours: 176 hours (22 work days x 8 hours)
Region: North Europe [prices vary per data center]
Pay as you go is yes
Compute payment options is monthly

Total cost for this set will be about 205.11 dollar for 1 virtual machine (you’ll often have 5 or more!):

Now scroll all the way down to SHOW DEV/TEST PRICING and flip this to yes. Now the cost comes at 169.73 dollar for a single VM, a 17.3% cost saving compared to non-discounted standard Azure pricing:

As you can see there’s no saving on the Disk storage which is a fixed cost which you’ll have to pay even when the VM is switched off. However, since the Virtual Machine is the highest cost, the discount will be higher the more hours the machine will be running. For example, when you select One year reserved equal to yes (which means full time continuous running for a year, similar to a sandbox), the total cost is 476.28 dollar as opposed to 744.92 dollar, a saving of 36%. Three years reserved (3 years of full time running) gives you a monthly cost of 331.20 dollar versus 599.84 dollar, a 45% saving.

2 – How to setup Azure and LCS to utilise the discounted pricing

When you open your the details for a Subscription on Azure portal, you’ll notice the Offer ID:

This Offer ID indicates the type of Offer that is related to a Subscription. Offers often either represent monthly credit or discounted rates. In the example above the Offer ID is a Partner Action Pack offering 85 Euro credit per month against non-discounted rates. You can find all available Offers here. Note that subscription can only have 1 activated at a time, but a tenant can have multiple subscriptions.

Two of these offers qualify for the discounted rates (see here):

Offer 0023P – Pay-As-You-Go Dev/Test: for customers who do not have an Enterprise Agreement
Offer 0148P – Enterprise Dev/Test: for customers who do have an Enterprise Agreement

Both these Offers provide discounted rates for:

Virtual Machines [D365 cloud hosted environments!]
SQL Database
SQL Managed Instance
HD Insight
App Service
Logic Apps

Here’s what you need to do to activate the discount for D365 F&O cloud hosted environments:

Create AAD Account: create an AAD account for the owner of the subscription who will also become the admin of the D365 F&O cloud hosted environments.
Visual Studio Subscription: Link a Visual Studio Subscription to this AAD account. A Visual Studio Subscription Professional comes at 45 dollars per month.
Activate the DEV/TST pricing: open the link to to the Offers mentioned above and click Activate. This will require you to login with the AAD account which you have defined to become the owner of the Subscription with discounted rates. Note: activation automatically creates the subscription in the tenant which relates to the AAD account. So you should be able to see the Subscription in Azure portal with the respective Offer ID attached to it.
Grant the AAD Account owner access to the Life Cycle Services (LCS) project where you want to deploy new Cloud Hosted D365 F&O environments.
Link the discounted Subscription to Life Cycle Services (LCS): follow the standard procedure described in this Microsoft docs page to link the new discounted subscription to LCS. Note: you can link multiple subscriptions to LCS.

Now when you deploy a new D365 F&O environment in your LCS > Cloud Hosted Environments, you’ll be able to pick the discounted subscription [ensure you’re logged as the subscription owner who will become the admin of the environment]:

Hopefully this discounted setup will allow you to enjoy working with the D365 family of Products even more 🎉!

Patrick

“Reflect upon your present blessings, of which every man has plenty; not on your past misfortunes, of which all men have some.” Charles Dickens

How to Unlock Many Hidden D365 Retail Features!

Mon, 11 May 2020 00:00:00 GMT

Ever since D365 Finance & Operations V10, we’re in the Managed Updates model which aligns all Customers on a similar Platform Version – It’s the so called “One Version” strategy which brings a lot of benefits. But is there really ONE VERSION in reality? We have License keys, Parameter modules, SysFlighting and Feature Management which already contains 250+ Features in V10.0.11 PEAP. Did you all enable them? If you sum all these parameters, D365 Finops installations can fundamentally differ even if the underlying Platform version is similar.

Now with D365 Retail (Commerce) there is yet another variable – A relatively new Retail/Commerce Parameter area which is controlling Retail functionality behind the scenes… In this blog post we’ll dig out all the Features that can be enabled from here. Fellow Functional and Technical D365 Retail/Commerce consultants sit tight: this area is likely to become more and important in the future. It can also be really helpful for your own customisations.

1 – Commerce Configuration Parameters – Introduction

If you open D365 Commerce and you navigate to Commerce Parameters > Configuration Parameters you end up in an area which seems rather boring:

But it’s actually the opposite: this is the place where unexpected magic can be unleashed. Maybe you remember the .INI files from the old days or the .config files in web applications – This is the place where you can set any parameter and parameter value in OPEN TEXT. So imagine you’re coding your own new Retail Feature and you’re implementing new magical Business Logic to enhance Omni-Channel Retail in 3 flavours: “Legacy”, “Full Retail” and “Ultimate Retail”. However, the “Ultimate Retail” flavour takes you months to build and you only want to enable it for a few special customers who will beta test it. It’s actually your little secret. In your X++ code, you then code a reference to evaluate a new Value MySecretRetailFeature in the Configuration Parameters grid with the 3 possible values “Legacy”, “Full Retail” and “Ultimate Retail”. Default is “Full Retail”, so all customers silently work by this setting. You don’t need to bother about implementing a new custom tab “My Customisation” with a new dropdown which is visible for all customers. Instead, you’ve just built your own secret Retail Flight Key (!).

If you look into the X++ code in D365 Commerce you’ll find Retail flight keys which are implemented by Microsoft behind the scenes. But we fellow D365 Commerce fanatics live in a special world: The D365 Retail/Commerce world. So OUR Flight Keys can even have more footprint as we have the ability to ship them to the Channel Database sourcing D365 Commerce Run Time (CRT). From there they can control behaviour of D365 POS, the Commerce Frontend and Retail API connected third-party applications. The shipment of the parameters happens through the RetailConfigurationParameters subjob which sits in the 1110 Global configuration Scheduler job:

In this blog post we’ll take a look at all the features which are exposed in this area, both in the D365 Backend as well as Commerce Run Time (CRT). Only a few of them are documented. Before we dive into more detail, here’s an example of a Retail Flight Key that is actually documented (Microsoft Docs):

FEATURE: Enable the ability to see Product Ratings and Reviews on the D365 POS
CONTEXT (Microsoft Docs): Product Rating and Reviews can be captured in the Commerce Storefront and made available in-store to serve the interaction between Store associates and Customers (for example aided by a dual screen POS).

This feature can be enabled by inputting the following Flight Key: RatingsAndReviews.EnableProductRatingsForRetailStores=true

2 – List of ‘Hidden’ D365 Retail/Commerce Features: D365 Backend

If you want to inquire all Retail Flight Keys developed by Microsoft which can be unlocked through the Commerce Parameters > Configuration Parameters form, then do the following:

Open the D365 AOT in Visual Studio
Search for “featurecontrol“
Select one of the classes -> Right mouse button -> View code
Scroll all the way up to the top of the code which lists the Flight Keys as constants:

See below an overview of all Flight Keys I could find in D365 V10.0.11 PEAP in applying this approach – Before you dive into this list please note:

Column “Default” indicates what the default setting is if the key is not set at all. In some place this is commented in code. Please note that the comment in code may be wrong compared to actual behaviour. I have not tested this (!).
Note that for Flight Keys that are used in the D365 backend, a string to int conversion is taking place when the Parameter value is read from the table. So to enable a Flight Key the word “true” is to be inputted and “false” for disabling. D365 Commerce Run Time (see section 3 in this blog post) expects integer values 0 or 1 (!). So this is also how you can distinct HQ vs Commerce Run Time (CRT) related Flight Keys :):

Overview | Retail & Commerce Flight Keys in the D365 Finance & Operations Backend

3 – List of ‘Hidden’ D365 Retail/Commerce Features: D365 Commerce Run Time (CRT)

As stated before, Retail Configuration Parameters (Retail Flight Keys) can be set in the Retail Parameters form in the D365 Finance & Operations back end and then shipped to the Channel Database by the 1110 Distribution Schedule. From there, D365 Commerce Run Time can use them to control operation in POS, Commerce Storefront or any other App. if you’re unfamiliar with the CRT Architecture principle, then I’d recommend you to read my blog post on the Basics of the D365 Retail and Commerce APIs. As will become clear below, Microsoft have already implemented the use of the Flight Keys in CRT in quite a lot of places and this is growing with every new D365 Release (!).

It’s pretty easy to find all the use of the Retail Configuration Parameters (the ‘hidden’ Retail Flight Keys’) inside CRT if you take my ‘Engine Room’ approach – For the Tech interested – Note: this is similar to finding an X++ reference for a table/form in the AOT:

Open .NET Reflector as described in the blog post and ensure to operate it against the latest CRT DLLs.
Navigate to Microsoft.Dynamics.Commerce.Runtime.Entities > Microsoft.Dynamics.Commerce.Runtime.DataModel -> RetailConfigurationParameter

Select RetailConfigurationParameters -> click right mouse button and select Analyze
Expand the tree structure and expand the child element Used By – Now all the references are listed:

I’ve inspected these references and listed the actual parameter key/value pairs below. Please note again that I have tested only some of these parameters.

Overview | Retail & Commerce Flight Keys in D365 Retail CRT

4 – Closing Words

Working myself down the lists above, I’ve discovered some really interesting Flight Keys – I’ve already applied some of them in my current project. Each of the Flight Keys is really a story in itself and testing can be time consuming. So please share your experience if you get to test any of them or find new ones in upcoming versions.

Note: While applying some of the Flight Keys I found a small bug in the use of the Flight Keys: some of the Flight key values exceed 60 characters while the Value column in Commerce Parameters > Configuration Parameters only allows for 60 characters. Even though this leads to truncation, the parameter is still picked up by D365 Finops. Microsoft is likely to fix the truncation in V10.0.13:

I hope this article inspired you to get even more out of you D365 Commerce implementation!

D365 Retail APIs Part III: How to use the Retail APIs from Power Automate (Flow) and Logic App

Thu, 02 Jan 2020 00:00:00 GMT

After walking through the overall D365 Retail API Architecture in part I and II of this series on the D365 Retail APIs, it’s now time to enable anyone to actually use the Retail APIs. To allow this, this blog post will contain all details on how to construct a request to most of the 400 out-of-box D365 Retail APIs – Beyond that, I’ve included a step-by-step instruction on how to use any of the Retail APIs from Microsoft Power Automate (Flow).

But before we can really get things rolling, we first need to apply some additional D365 setup and we need to choose the right security pattern. So that’s where I’ll start this blog post.

Note: to effectively consume the content of this blog post, please ensure to familiarise with the content of Part I and Part II.

1 – Security

In Retail world, we’re utilising a variety of sales channels which are operated by a variety of users: in general, 3 types of users can be recognised:

Employees: in-store POS systems and Call Center applications are operated by Retail employees.
Named consumers: consumers unveiling details on their identity (name, address etc.) while making purchases via Social media, e-com platforms, consumer apps etc.
Anonymous consumers: guests making purchases without consent on saving information about their identity

Looking at privacy and data security in general, it is crucial to restrict access to the 400+ D365 Retail API end points for use by each of the 3 categories of users. For example, we certainly do not want the guest visitors on our e-com website to be able to retrieve information from other customers on loyalty points, purchasing history or product recommendations. At the same time, we don’t want our self-checkout consumers to apply manual discounts :). In summary, we need a mechanism to make our Retail end points (which are the gateways to our Data and Business logic) available to only those users who are allowed to use them.

If you look inside Retail Server as we’ve done in Part II, you’ll find that each of these end points have a special attribute called “CommerceAuthorization” (see picture 1 below). This attribute basically lists the 1 or multiple “Commerce Roles” which are allowed to use the endpoint.

Picture 1 – CommerceAuthorization Attribute on a Retail Server OData endpoint

These “Commerce Roles” are very much in line with the user groups we acknowledged earlier:

Role 1 – Anonymous: this role represents our anonymous customers. It will be of no surprise that only a few D365 Retail APIs are exposed to be used anonymously
Role 2 – Customer: this role represents our named consumers
Role 3 – Employee: this role represents our Retail staff
Role 4 – Application: this generic role is designed for third-party applications which do not operate on behalf of a specific user (Customer or Employee)

Picture 2 below depicts the purpose of Commerce Roles in a bit more detail. As you can see in this picture, every D365 Retail API is enabled for 1 or multiple roles. Since the Customer and Employee role are built around a known identity (from a consumer or staff member respectively), both roles build upon the use of ID Tokens. In short, ID tokens contain encrypted details on a person’s identity (name, account number etc.) In case of Employees (in-store!) the encrypted ID Token contains the Employee number and Password which we know are used for logging into MPOS/CPOS.

Last but not least, picture 2 highlights the Commerce Role Application. This is the role we’ll focus on in the remainder of this blog post as it can be used for integrating any third-party application (e-com platforms, third-party POS systems, Apps etc.). As visible in picture 2, the Application role works by Access tokens. This is exactly the type of authentication most of you will recognise from the very commonly used D365 F&O (backend) APIs. Remember: similar to the D365 F&O backend APIs, the D365 Retail APIs are also based on OData, so there’s not much of a difference on how to consume them as you will see below.

Picture 2 – The purpose of Commerce Roles

2 – Azure Setup

In order to allow an external application to make requests to the D365 Retail APIs, we need to register the application in our Azure Active Directory (1) and allow access to D365 for this application (2). Let’s start with the Azure side of things. Here’s how to register the application:

STEP 1 – Register the Application

Open Azure portal
Navigate to Azure Active Directory > App Registrations
Click New registration
Enter the following details (note: Name can be the name of your application such as the name of the e-commerce platform, App etc.) and save these details

STEP 2 –Create and Copy App Secret

Navigate to Azure Active Directory > App Registrations -> Click on the App you registered in step 1
Click Certificates and secrets
Create a new client secret as per below picture
Select the Copy to clipboard icon to copy the Secret
Paste the Secret in a safe place, for example a OneNote page – You’ll need it later

STEP 3 –Create and Copy App Secret

Navigate to Azure Active Directory > App Registrations -> Click on the App you registered in step 1
Click Overview
Copy the Application (client) ID
Copy the Directory (tenant) ID
Paste both IDs in a safe place, for example a OneNote page – You’ll need them later

3 – D365 Setup

STEP 1 –Register the Identity Provider in Retail Shared Parameters

Navigate to Retail > Channel Setup > Channel Profiles
- Copy the Retail Server URL without the /Commerce suffix
- Keep a note of this URL, for example on a OneNote page – You’ll need it below
Navigate to Retail > Heaquarter Setup > Parameters > Retail Shared Parameters -> Tab Identity providers – Create the entries as shown in the picture below:
- IDENTITY PROVIDERS > Issuer: Please note the / behind the URL so https://sts.windows.net// – Paste the tenant ID as copied in Azure Setup -> Step 3 above
- RELYING PARTIES > ClientID: paste the Application ID as copied in Azure Setup -> Step 3 above
- SERVER RESOURCE IDS>Server Resource ID: paste the Retail Server URL which you copied from the Channel Profiles form

STEP 2 –Assign security permissions to the Application

Navigate to System administration > Setup > Azure Active Directory applications
Create a new record and paste the Application ID as copied in Azure Setup -> Step 3 above
Select a user account for the Application – Please note that the user should have sufficient privileges for the operations the external Application should be able to perform against D365.

STEP 3 – Synchronise the Setup to the Retail channel Database

Navigate to Retail > Retail IT > Distribution schedule
Run the 1110 (Global configuration) schedule
Note: in some rare cases I’ve come across it was needed to run the 9999 job

4 – Request format

Now the Setup on Azure and D365 is done, we’re ready to make the D365 Retail APIs work for us.

STEP 1 – Request an Access Token

Request format:

Notes:

{{Tenant ID}}: paste the Tenant ID as copied in Azure Setup -> Step 3 above
{{Azure Application ID}}: paste the Application ID as copied in Azure Setup -> Step 3 above
{{Application secret}}: paste the Application secret as copied in Azure Setup -> Step 2 above
{{Retail Server URL}}: paste the Retail Server URL as copied in D365 Setup -> Step 1 above
The access_token value from the response is to be passed as Authorization header value in the actual request (see step 2 below)

STEP 2 – Make the Request to Retail Server – Example: create a customer into D365 backend via the Retail APIs

Request format:

Notes:

{{D365 Retail URL}}: paste the Retail Server URL as copied in D365 Setup -> Step 1 above
{{access token}}: copy/paste the value from the access_token entry in the Body response from step 1
{{retail channel operating unit}}: copy/paste the Retail channel number representing the Store or Online store. You can find this number on the Organization administration > Organizations > Operating units form in D365 F&O. For example: 052 represents the Houston store.

5 – Postman Setup

To kick-start you on using the D365 Retail APIs, I’ve packaged above Request into a ready-to-go Postman setup. Follow below steps to get it rolling.

STEP 1 – Download and Installation

Download and install the Postman client
Download the Postman Environment and Request Collection I have prepared for you from my OneDrive.
Open Postman and click File>Import – Click Choose files and select the 2 files as downloaded from my OneDrive: D365 Retail APIs Part III – Environment.postman_environment and D365 Retail APIs Part III – Requests.postman_collection

STEP 2 – Fill out the Environment variables

In the top right, set D365 Retail APIs Part III – Environment as the active environment
Click the Manage environments icon
Click on D365 Retail APIs Part III – Environment
Fill out the values which are specific to your environment (See Azure Setup and D365 Setup above) for the entries starting with Input your..
Click Update to save the new values

STEP 3 – Create a Customer via the Retail APIs

Expand the Postman collection D365 Retail APIs Part III – Requests
Select the POST Get AAD Token operation
Click the Send button on the right hand side
Verify you have a Body response which is similar to the response in the picture below

Expand the subfolder D365 Retail APIs in the D365 Retail APIs Part III – Requests Postman collection
Select the POST Customers\CreateCustomer operation
Click Body on the right hand side and optionally update the values such as the name for the customer
Click the Send button on the right hand side
Verify that the Body response returns all the details for the account which was newly created – You’ll find this account in the Accounts receivable>Customers>All Customers form in D365 F&O
When inquiring the new Customer account you’ll find out that many properties such as the Sales tax group, Address book, Credit limit etc. are all populated while you did not input them. This is the heavy lifting done by the standard Retail Business logic which is sitting behind the Retail API. As per this logic, many properties are inherited from the default customer account as configured on the respective Store or Online store. This is a big advantage of using the D365 Retail APIs as over the standard D365 F&O Data Entities: if you use these, you’ll have to fill out almost every single property for the customer which is very time consuming (as it’s “re-inventing the wheel!”). With the Retail APIs, Microsoft have done most of the ’thinking’ for you which also guarantees you’re 100% compatible with the path the D365 Retail product is heading.
Note: the first time a request to Retail Server is fired in a DEV box, it needs to ‘warm boot’, so the request will take a couple of seconds as opposed to the normal duration of about 150 milliseconds.

6 – Using the D365 Retail APIs with Power Automate (Flow)

Now we know how to consume the Retail APIs which have the Application CommerceRole (read: are suitable for interaction with third-party applications), I’ll know show you how to directly consume the D365 Retail APIs from Microsoft Power Automate (Flow). This will enable you to easily use the Retail APIs in Microsoft Power Apps and will simplify your D365 Retail integration experience.

In my first blog post on the D365 Retail APIs I included this video which demonstrates the use of the D365 Retail APIs by creating customers into D365 F&O from an Office 365 Form. Below I’ll show you step by step how to set this up in Office 365 Forms and Microsoft Power Automate. Here’s an overview of the Architecture:

Part 1-3 of this Architecture will be detailed in the steps below. Part 4 and 5 have been detailed in previous blog posts on this topic. Please note that the Office 365 form is linked up with the D365 Retail APIs via 2 Flows as opposed to 1: I have separated the Flow steps which are to request the Access token and to fire the request to D365 into a separate Flow (the Abstraction Flow), so this can be re-used to work with any other client (other than the Office 365 form in this example). Please be aware that we first have to create the Abstraction Flow (Part 3) so we can call it from the Orchestration Flow (Part 2).

STEP 1 – Create Office 365 Form

Navigate to Office 365 Forms: https://forms.office.com/Pages/DesignPage.aspx
Click New form
Create the form as per the picture below – Please ensure the dropdowns/radio buttons for questions 3-5 actually reflect the values in the subtitle for the question (these match with the D365 Contoso data set)

STEP 2 – Create the Abstraction Flow

Navigate to Microsoft Power Automate: https://emea.flow.microsoft.com/
Click Solution>New Solution and input and save details for the new Solution
Click New>New Flow to start a new Flow – name this Flow Retail API_Orchestration
Build and save the following Flow:

{
"type": "object",
"properties": {
"Phone": {
"type": "string"
},
"Email": {
"type": "string"
},
"FirstName": {
"type": "string"
},
"LastName": {
"type": "string"
},
"CustomerGroup": {
"type": "string"
},
"CurrencyCode": {
"type": "string"
},
"Operating Unit": {
"type": "string"
}
}
}

Input the value for your Azure Tenant ID

Input the value for your Application Secret

Input the following formula: replace(replace(variables(‘Secret’),’+’,’%2B’),’=’,’%3D’)

Input the value for your Azure Application ID

Input your Retail Server URL

{
"type": "object",
"properties": {
"token_type": {
"type": "string"
},
"expires_in": {
"type": "string"
},
"ext_expires_in": {
"type": "string"
},
"expires_on": {
"type": "string"
},
"not_before": {
"type": "string"
},
"resource": {
"type": "string"
},
"access_token": {
"type": "string"
}
}
}

{
  "Phone": "@{triggerBody()?['Phone']}",
  "Email": "@{triggerBody()?['Email']}",
  "FirstName": "@{triggerBody()?['FirstName']}",
  "LastName": "@{triggerBody()?['LastName']}",
  "CustomerTypeValue": 1,
  "Language": "en-us",
  "CustomerGroup": "@{triggerBody()?['CustomerGroup']}",
  "CurrencyCode": "@{triggerBody()?['CurrencyCode']}",
  "ReceiptSettings": 0,
  "ReceiptEmail": "@{triggerBody()?['Email']}",
  "Addresses": [],
  "Attributes": []
}

STEP 3 – Create the Orchestration Flow

Return to the Solution you created earlier
Click New>New Flow to start a new Flow – name this Flow Retail API_Office 365
Build and save the following Flow:

Note: select the Office 365 Form that you created in step 1

After saving the Abstraction Flow in Step 2, you can copy the URI (URL where this Flow is hosted) – Input this into the URI field
Note: do not forget Content-Type=application/json – It is mandatory when calling an endpoint which represents another Flow

{
  "Phone": "",
  "Email": "@{outputs('Get_response_details')?['body/responder']}",
  "FirstName": "@{outputs('Get_response_details')?['body/reba91af4c50648c4a9a872a441d9a4be']}",
  "LastName": "@{outputs('Get_response_details')?['body/r733d0622029b45e8a860d22fac487e79']}",
  "CustomerGroup": "@{outputs('Get_response_details')?['body/r66eb983cdf354a2ca941b3fc6ac457f4']}",
  "CurrencyCode": "@{outputs('Get_response_details')?['body/r4d4605397f064fb491135d22fc4abcda']}",
  "Operating Unit": "@{outputs('Get_response_details')?['body/r220e4dbf5e4445628734a9c249494375']}"
}

7 – Final Words

I hope the length of this blog post did not discourage you to get started using the D365 Retail APIs. I can tell you from experience that when you reach the point that you’re getting the feeling of Mastering the Retail APIs, you’ll be blown away by the Power and endless Opportunities of this Treasure. You’ll wake up with new ideas on how to use them everyday. Power Apps, Power BI, Power Automate, Power… RETAIL!

D365 Retail APIs Part II: How to know exactly what happens inside D365 Retail

Fri, 13 Dec 2019 00:00:00 GMT

In any of our D365 Implementation Projects, Solution Prototyping is one of the most important aspects of the implementation. On the one hand this Prototyping serves a common Requirement to find a solution which stays closest to Standard D365 – A Requirement often raised by small to medium sized Retailers. On the other hand, Prototyping let us fully understand D365 Retail which allows us to write the best designs for Custom solutions – An exercise which is often required for the bigger Retailers.

Solution Prototyping is most commonly done from ‘Sea level‘: by setting up D365 Retail in different ways to support variations to a Process which are then evaluated (often in playbacks to Business key users) against Process maps and related Requirements. I think this is a very good and effective way to make progress in a project while still making the right Design Decisions.

However, there are cases in which this ‘Sea level’ approach is either very time consuming or not giving you the confidence that you’ve completely bottomed out the D365 Retail capabilities to support a certain process. You probably recognise this from working on complex Processes which are Retail/Omni related, D365 MPOS/commerce customisations or Retail Integrations. For these cases, you need another approach which I’ve named the ‘Engine Room‘ approach: this approach first descends into the ‘Engine Room’ of D365 for Retail to know exactly what happens there before you work yourself up to ‘Sea level’ again to pick the best Solution for your Use Case. In this blog post, I’ll unveil all the tricks to be able to do this quickly and effectively.

Note: please ensure to familiarise with the D365 Retail API Architecture as per Part I of this series on D365 Retail APIs before you read any further.

1 – Approach

Picture 1 below depicts my ‘Engine Room’ approach:

Trick 1 – IIS Setup and Fiddler : we’re going to make a modification to the IIS/authentication to allow to intercept all traffic between all D365 Retail IIS Services with Fiddler – See this picture on the D365 Retail supporting IIS services you’ll typically find on a D365 F&O DEV box.
- A. The D365 Commerce and D365 SharePoint Online store >> TO >> D365 Retail Server
- B. D365 Cloud POS (CPOS) to >> TO >> D365 Retail Server (note: this can also be done without the trick – see blog post by our Retail Champion Kurt Hatlevik here)
- C. D365 Retail Server >> TO >> D365 Real Time Service (RTS)
Trick 2: .NET Reflector: we’ll install .NET Reflector to be able to look inside D365 Retail Server, Commerce Run Time (CRT) and all related portable Retail Business Logic
D365 AOT: we’ll use Visual Studio/AOT (no tricks needed here :)) to look into the D365 backend Real Time Service class logic hit by D365 Retail Server
D365 X++: Debugging: optionally we can debug the D365 code beyond the RTS classes to know what’s happening further down the road in D365 (Business as usual ;-))

Picure 1: the ‘Engine Room’ approach

2 – Setup

2.1 – Trick 1: IIS Setup and Fiddler

Note: only a apply this trick on a D365 DEV box!


#	Step	Reference
1	RDP into the D365 VM	Use the admin account + password available through LCS
2	Follow these steps to install (if not already installed) and configure Fiddler on the VM.	Link
3	Open Internet Information Services (IIS)
4	Navigate to the Application Pool for Retail Server > Advanced settings -> Change Identity from “Network Service” to the admin account (requires password to be entered). Note: this is the admin account which is available for the environment on LCS and which is used to RDP into the environment	IIS Settings
5	If you’re working with CPOS, this is sufficient to be able to intercept all end-to-end CPOS>Retail Server>RTS traffic.	If you’re working with the SharePoint or Commerce storefront, then repeat step 4 for the Application Pool representing the storefront
6	I’d recommend to reboot the environment after above steps

2.2 – Trick 2: Install and Setup .NET Reflector


#	Step	Reference
1	RDP into the D365 VM	RDP with the admin account + password available through LCS
2	Download and install .NET Reflector (run as Administrator)	Link Note: this tool is not free, but has a 14-day trial – I prefer the standalone edition – There’s also an Add-In for Visual Studio
3	Open Internet Information Services (IIS) – Navigate to Sites and select RetailServer. Click right mouse button > Explore. This will open Windows Explorer for the folder which hosts the RetailServer website
4	Double click on the Bin folder and copy the path to the folder	e.g. K:\RetailServer\WebRoot\bin
5	Open .NET Reflector and click File > Open Assembly – Paste the path to the RetailServer folder as copied in the previous step into the File name textbox. Click Open to open the path
6	Select one of the DLLs and click to select all DLLs – Click Open to open the DLLS
7	Navigate to the following entry: `Microsoft.Dynamics.` `Retail.RetailServer\` `Microsoft.Dynamics.` `Retail.RetailServerLibrary\` `Microsoft.Dynamics.Retail. RetailServerLibrary.ODATAControllers`. This entry represents the Retail Sever “External mailbox” (Retail OData end points) – You’ll recognise all Retail operations here! This will always be your starting point for any analysis, so keep this entry in mind (!)	D365 Retail OData endpoints

3 – Analyse a Real-life example: create a Customer through the D365 Retail APIs

In Part I of this series on D365 Retail APIs I used Microsoft Forms to create a Customer into the D365 backend utilising the D365 Retail APIs in real time. Now, we’ll dive into the ‘Engine Room‘ of for this process of customer creation. As you’ll see this will unveil all aspects of Customer sign up including how the Retail Setup and Parameters in the D365 backend guide this process.

3.1 Intercept the D365 Retail communication with Fiddler

Ensure to have Fiddler and .NET Reflector open (open as Administrator).
Fire up a D365 Retail Cloud POS and navigate to Current Transaction > Add Customer > Create new Customer
Clear all sessions from Fiddler
Fill out the customer details and hit the Save button
Wait until the customer is fully saved and then open Fiddler – Inquire the following results:

CPOS >> TO >> Retail Server communication

Retail Server >> TO >> RTS communication (note: this version D365 V10.0.8 PEAP)
We can derive the following information from the Fiddler input:
- CPOS is making a request to /commerce/customer in JSON format
- Retail Server is making a request to /Services/RetailCDXRealTimeService in XML format
- RTS The SOAP service forwards the request to the RetailTransactionService.NewCustomerExt2 method in D365 X++

3.2 Analyse what happens within Retail Server

Now we have all the entry points for analysis, let’s dive into related Retail Server Business Logic underpinning the request from Cloud Pos to Retail Server:

Open .NET Reflector and the Retail Server DLLs as explained in paragraph 2
Navigate to the Customers OData Controller:
Now let’s follow the ’trail’ 😉 – Click on this.customerManager.CreateCustomer – This will bring us into the Commerce Run Rime (CRT) area.
As you can see here, a request is created to a Workflow (a series of Retail Services) – From here we can inquire all ‘Customer Create’ Business Logic as sitting on D365 Retail CRT
We can derive the following conclusions from inquiring this Business Logic:
- If the Channel is an Online store then Retail Server is reading the setting whether to create an Asynchronous (offline) customer from the Online store -> Functionality profile
- if the Channel is a Store then Retail Server is reading the setting whether to create an Asynchronous (offline) customer from the Retail store -> Functionality profile
After determining whether to create an Offline customer as per the previous step (in Contoso: No=create a customer in Real time), the CRT Workflow fires a saveCustomerServiceRequest: if the Customer Account is to be created Asynchronously (offline), then a customer Account number is generated locally and the customer is created into the local Retail database via a Data service (eventually a stored procedure). If the Customer Account is created Synchronously (real time), then a request is fired to D365 backend RTS. As we already know from our previous Fiddler analyses, this is the flow that is followed for the Houston Store (our example here) in the standard Contoso data set. Fiddler showed that the actual D365 method that is hit in the D365 backend is RetailTransactionService.newCustomerExt2. Let’s take a look in there in the next section!

3.3 Analyse what happens within D365 (Retail) Backend

Open Visual Studio -> D365 AOT on the D365 DEV Box
Navigate to RetailTransactionService.newCustomerExt2 – As we can read in the comments for this method, this method also supports Extension properties (=custom customer properties in CPOS/Retail Server in case it’s not sufficient to work with the standard Customer Attributes)
Let’s follow the ’trail’ (call stack) from here into RetailTransactionServiceCustomers.newCustomerExt -> RetailTransactionService.newCustomer -> RetailTransactioServiceCustomer.newCustomer
Here we find a treasure of Retail Business Logic for creating a customer into the D365 backend – For example, many customer properties are defaulted from the Standard Retail customer which is configured on the Respective channel.
Feel free to follow the trail deeper into D365!

4 – Some last words…

For the ones who are ‘lost in the woods’ after reading this long blog post, see below an overview of the full E2E ‘Call stack’ for creating a Customer into D365 backend in real time… Yeah.. this WHOLE process happens under 200ms once the Retail Server and Real Time Service are ‘warm booted’… I think it’s simply amazing. What a master piece of art Microsoft have created here !

I hope this blog posts give you a kick start in performing any ‘Engine Room’ analysis against D365 Retail. I always learn a lot on how things are really done inside D365 retail when I follow this approach!

The Power of the D365 Retail APIs/CRT – Part I: The Basics

Fri, 06 Dec 2019 00:00:00 GMT

Did you know that D365 for Retail contains more than 400 out-of-box Retail APIs (for the non-techies: interfaces) ranging from Loyalty management, Gift card management, Omni order management to Stock counting and Retail Logistics? These interfaces allow you to easily integrate with any third-party systems like e-commerce platforms, POS systems and Consumer Apps. It also allows you to quickly enrich your Retail Solution with handy Power Apps, for example for in-store Goods receipt, Stock counting, Labelling or Clienteling.

Unfortunately I still notice that many Customers and Partners do not consider the Retail APIs in their designs, which goes around all the ‘free’ Retail Business Logic which comes with them. Instead, Customers and Partners start a custom coding exercise (re-inventing the wheel!) which often limits rather than empowers the omnichannel capabilities of D365.

This is all mainly caused by the lack of documentation – It’s time to change that! In this series of blog posts we’ll explore the Power of the Retail API’s so we can all benefit from them by using them to enhance the Retail experience for the Retail Businesses we serve. As we need the non-techies on this journey, I’ll try to make it understandable for all of us. Making the right Architectural decisions starts with a good understanding of the solution. So let’s dive into that first!

1 – D365 Retail API Architecture explained

D365 Retail comes with 3 major components – Below description is a little bit technical, but it’s good to keep these components in mind for a better understanding of the main picture (picture 1) below:

Let’s go through each element in the picture, step-by-step:

The Letter. An app (e.g. a POS system, e-commerce platform or Power App) can send a request to 1 of the 400+ D365 Retail end points. This request can be to Request information such as the loyalty balance of a customer or to Send information.
The public Mailbox. Metaphorically you can compare this request with a letter which is dropped into a mailbox. The mailbox in this case is the OData endpoint of Retail Server (1A). It’s a public mailbox, so any App can drop letters in there. But there are 2 prerequisites which need to be met to allow this: the App needs to be a registered ‘member’ in the Receiver’s domain and the App must have received a pre-authorisation code (token) which needs to be sent with the letter.
The Internal Mailbox. Once successfully received in the public (OData) Retail Server mailbox, the letter (request) is forwarded to an internal mailbox: a specific Commerce Run Time (CRT) operation (1B). This CRT operation is responsible for processing the letter. In order to get this done, the CRT operation contacts 1 or multiple Retail Services (1C).
Processing of the letter. The Retail Services work with subcontractors. There are 2 types of subcontractors: Data services which can read/write information from/to the local Retail Database or Remote Data services which can receive/send information from/to the D365 backend. The Retail Services have been instructed (programmed) to contact either of the subcontractors or both subcontractors to process a certain letter (request). For example, a letter to update the Loyalty Balance for a customer is forwarded to the D365 backend (since other Apps should work against the latest balance) whereas a new customer is created both in the local Retail Database and forwarded to the D365 Backend. In some case the Retail Service has to read specific parameter settings first to determine which subcontractor to forward the work to. These parameters have been set in D365 backend but are locally available for the Retail Services after sync to the local Retail Database. For example, a parameter which controls whether e-com orders are created into the local Retail database (and from there synched to D365 backend in bulk) or whether these orders are forwarded to the D365 backend in real time.
Another Public mailbox. Remote Data Service ‘Subcontractors’ will drop the original letter in the Public mailbox of the D365 Backend: a SOAP mailbox, often referred to as Real Time Service (2A). This mailbox has a trust relationship with the Remote Data Service ‘subcontractor’ which is confirmed by a certificate.
Another Internal mailbox. Very much similar to the way Retail Server is organized, the agent looking after the Real Time Service mailbox, forwards the letter to an internal mailbox for processing: the RetailTransactionService class in the D365 Backend (2B). From there, Retail Services in the D365 backend take care of the actual processing which ties into many standard D365 (Retail) classes packed with Business Logic (2C and 2D). For example in case of customer creation, these services will include the customer in the address book of the Retail channel, assign Retail affiliations and Customer attributes. In other words: even though not all this information was part of the original letter, the Retail Services ensure the letter is processed in line with D365 (Retail) Parameters settings and Retail Business Logic.
Response. As a sender of a letter, you’d love to get a response on your respect, right? This is organized as follows: whenever an exception occurs somewhere across the chain of ‘middlemen’ subcontractors as described above, the exception is written into a response letter which is returned to the sender (the App) following the chain in reverse order. If no exceptions occur, the original letter will reach it’s final destination. There, a response letter is written which is an in-depth response on the actual request, for example a response letter full of information on a newly created customer. Similar to what happens in case of an exception, the response letter is returned to sender following the chain in reverse order.

2. Teaser

In below video I am trying to illustrate how powerful the Retail APIs can be. In the video, I am using Microsoft Forms (an Office 365 App) to capture information as part of an imaginary Customer Sign up process. Power Automate picks up the information and forwards it to D365 Retail Server to create the customer in the D365 backend in real time. As you will see in the video, the standard D365 Retail Business logic is fully utilized here as if the customer was captured on D365 MPOS: for example, the new customer is automatically included in the Retail channel’s address book.

3. The potential

The left Table below provides a break down of the Retail areas the 400+ Retail APIs apply to. A significant number of these APIs are built to provide real time data exchange with the D365 backend. The right table below presents the Retail areas for which real time connectivity to the D365 backend is supported.

Looking at the functional breadth of the Retail APIs and the video in this blog post, I hope you’re inspired to think about all the opportunities provided by the D365 Retail APIs. In upcoming blog posts on this topic, we’ll make deeper dives into the various aspects of these Retail APIs so we can all use this knowledge to further enrich the D365 Retail experience for our customers.

Advanced RSAT I: How to run D365 RSAT from MS FLOW

Thu, 26 Sep 2019 00:00:00 GMT

We’re now in the new world of D365 “OneVersion” – A world in which we can benefit from continuous Microsoft investment without having to re-implement ERP ever again. But the new world also requires a solid Regression Testing Strategy to control the cost for consuming new updates. Here’s where Microsoft stepped in giving us the Regression Suite Automation Tool (RSAT). Although this tool is a good starting point, anyone who ever worked with RSAT will probably recognise one or more of the following limitations:

I don’t want to go into a VM to run test cases!
How can I test my D365 interfaces???
How can I run multiple test cases/test suites in parallel (in other words: multi threaded)??
I don’t want my chained tests to be scripted in Powershell, but in Azure DevOps (low code/no code solution!)
How can I run mixed tests of process and interface? – e.g. to support E2E scenarios like: my e-com platform creates a sales order through an interface into D365 which is Released to warehouse through RSAT?

We can of course wait for Microsoft to come up with solutions to overcome these limitations. In my case, I decided to accept the challenge to leverage the Power Platform to fill the gaps. In this series of blog posts I’ll gradually tackle each of the above limitations one by one.

Key step in the overall solution is to find a way to move away from executing our D365 test cases directly from the RSAT tool and to bring this into a higher control layer with broader capabilities than RSAT alone… Please welcome an RSAT connector for Microsoft Flow!

Get started with the QuickStart Guide I published on my OneDrive:

https://patrickmouwen-my.sharepoint.com/:b:/p/contact/Ebnvx0OR_mFPsnMqHZBFQN8BiAmehDVTl7oeCpLrdnHHsw?e=xwmi45

Advanced RSAT II: How to run D365 RSAT directly from Azure DevOps

Wed, 25 Sep 2019 00:00:00 GMT

Now that we’ve constructed a ‘RSAT connector for Microsoft Flow’ in Part I of this Series, we have lots of new opportunities offered by the Microsoft Power Platform to push our Test Automation in D365 to a new level.

In below video I’ll demonstrate how toggling the State of an Azure DevOps Test Suite to “Run RSAT” triggers MS Flow to iterate through the Test Cases in the Test Suite and let RSAT execute the Test Cases (D365 Task Guides) in the background. MS Flow will wait until RSAT execution is finished to send an e-mail to notify that RSAT execution is done. The e-mail will have the logs from the RSAT execution as attachments.

See my OneDrive for the Flow which is used in the video.

Note: I didn’t manage to import this Flow package into another tenant than the tenant where the Flow was originally built – MS Flow complaints about a problem with the authorisation against Azure DevOps – So you may have to amend the packaged (zipped) json definition of the Flow to allow import. Please share an updated .zip file if you manage to make amendments to the packaged .json definition which allows import.

Identify Configuration differences across D365 F&O environments

Tue, 11 Jun 2019 00:00:00 GMT

We’ve all seen it many times in our projects.. Being under pressure to close these last P1 and P2 bugs by making some quick config changes in UAT, we forget to keep a record of what we did and to bring this into our Golden Config environment. And then, after cutover to Production, our users are confronted with the same issues as before.. How great would it be if we can prevent this with an extra ‘lock on the door’?

Dynamics Versions

The method of comparing configuration between D365FO environments as described below is utilizing D365 Data Management capabilities. If you’re still working with AX2012 you’d also be able to apply most of it with DIXF.

How it’s done

STEP 1: Represent your critical configuration areas in a D365 Data Template

We all know our ‘sensitive’ setup areas which can give real trouble when not 100% identical in the various D365FO environments – Areas such as Financial dimension values, Tax setup and module parameters. To have a good starting point for your Config ‘lock on the door’, ensure to represent these areas by Data Entities in a D365 Data Template which you could for example name “D365_Setup_Compare”:

Export this template into a SharePoint library for easy re-use and version control.

STEP 2: Create Export Data Projects in your D365FO environments

Create an Export Data Project on the basis of the Data Template from SharePoint and pick “XML Element” and “Data Package” for the output format. As per best practice, suffix the Data Project with the name of the environment for easy identification later. Note that in case you have many D365 FO environments to continuously apply updated Data Templates to, you could consider Data Task Automation.

STEP 3: Save the files in a central location

Create a folder structure in a central location. If you use SharePoint as a baseline, then open the respective Document Library in OneDrive to allow sync to local folders. Then, run the Export Data Projects in the various D365FO environments, download the packages and extract them into the folder structure:

STEP 4: Install and configure Altova diffdog

To quickly compare Configuration from D365FO environments by the XML files we’ve exported, I prefer using a tool called Altova diffdog. Although the tool is not free (one-time EUR109/USD139 for a license), I think it’s worth the investment looking at the excellent way this tool highlights any differences at a glance – We’ll come to that below. First, download and install the tool (on a trial license) and configure as follows:

Click File>Compare Directories and select the directories to compare
Tick Compare as XML and untick Quick to let Diffdog perform a detailed comparison of the files in the directories – This is done in seconds
The tool then highlights the differences as per color coding as you can see in the example below.
Click File>Save as and define a file name which is representative for the comparison, such as “D365 UAT vs PROD”. In this way, you can have multiple comparisons open as different tabs in Diffdog.

A click on any of the colored rows which represent differences, will give you the details. Switch to Grid View for the best experience:

Further ideas

If you want to make this a fully automated process which is re-usable for any other project, I’d recommend the following:

Utilise D365 Data Task Automation to automate the creation of “D365_Setup_Compare” Data Projects and their Data Entities representing core Setup Areas to compare. In this way, you can centrally maintain your project definitions in your LCS Shared Asset Library for easy re-use in all your Projects.
Introduce automatic updates of the XML files in the SharePoint folder by setting up D365 Recurring Data jobs for the “D365_Setup_Compare” Data projects and configuration of a Microsoft Flow to automatically dequeue, download and unzip the content of the packages into your central SharePoint folder.

Hope this ‘lock on the door’ saves us all a lot of frustration..!

Patrick

Dynamics 365 CRM and ERP interaction

Thu, 28 Dec 2017 00:00:00 GMT

Overlooking many recent Dynamics 365 implementations, it seems that Microsoft’s marketing strategy of ‘marrying’ Dynamics AX and CRM into one Dynamics 365 family is paying off. More and more customers are choosing for the package deal of Operations and Sales. However, as always, the marketing machine is ahead of the troops: with the current state of the integration of the products, we need to be creative in order to make the two systems feel like one.

In this blog post I’ll unveil an effective way to let Microsoft Dynamics 365 for Sales (CRM) interact with Microsoft Dynamics 365 for Finance and Operations (ERP)…

Introduction

Most of us will have some affinity with the Common Data Service, which can also be used to exchange data between Dynamics 365 for Sales (CRM) and Dynamics 365 for Finance and Operations (ERP). However, first of all Common Data Service does not offer real time capabilities at the moment of publishing this blog post. Second, this information exchange happens behind the scenes (on data/service level). But in order to make D365S and D365FO feel like one system, we need a different kind of interaction. An interaction which builds upon real timeexchange of information and which builds upon data interaction on form level: creating a new D365FO sales order from D365S should offer an experience which is close to ‘native’ D365FO experience.

Concepts for CRM and ERP interaction

In a recent project context, the customer was looking to emphasize D365S as master system for Customer management and Customer interaction and to emphasize D365FO as their selling system. Therefore, all users were envisioned to work from a 360 degree customer dashboard in D365S which was also CTI enabled (=telephone integrated). Customer interaction in D365S could result in offering a quotation or direct sales which then required to bring the user into the D365FO sphere. In order to make this a seamless experience for the user, we evaluated two concepts:

Hub-2-Hub: in this concept, the user has one “Open in Call center” button which opens the D365FO Customer Service form for the Customer Account currently selected in D365S. This Customer service form then offers the out-of-the-box D365FO features to create or edit an existing sales order or return order (quotation was added by minor modification).

See this concept in action (example):

Master-Slave: in this concept, the user working from D365S has direct access into the D365FO quick create forms for creating new sales orders and quotations and into the D365FO details forms allowing direct modifications to sales order or quotation. In this project context, we also decided to map the D365FO order holds to D365S tasks. This allowed Call Center agents to spot any order exceptions in D365S (for example missing payments), tackle that exception with the customer and then remove the respective hold by one click into D365FO.
See this concept in action (new order):

See this concept in action (new quotation from opportunity):

In both concepts, the data modified or newly created in D365FO is synched back to D365S through the standard CDS templates currently available. So after some minutes, the 360 degree customer dashboard in D365S reflects these changes at next visit of the customer Account. In a few scenarios, the current non-real time CDS behavior was not sufficient to support the business scenario: for example, new customer accounts created in D365S are required to be available in D365FO straight away. Otherwise the quick create form for quotation or sales order creation in D365FO cannot be pre-filled for the customer which was just created in D365S (Master-Slave concept) and the Customer Service form in D365FO cannot be pre-filtered for the that new customer (Hub-2-Hub concept). So here we had to implement real time Azure LogicApp integration to get to the desired result (no further details in this blog post).

The trick: how to realize the interaction on form level

The ’trick’ to make D365S to D365FO form level interaction possible is to utilize the relative freedom we have to leverage custom URL parts (suffixes) in D365FO. A typical D365FO URL for opening the Sales order form looks like this: https://.sandbox.ax.dynamics.com/?mi=MCRCustomerService&cmp=

This is the URL that was opened from D365S in the Master-Slave\Create quote scenario:
https://.sandbox.ax.dynamics.com/?mi=MCRCustomerService&cmp=&customerid=575342

As you can see, both the D365S Account number (kept in synch with D365FO Customer account through LogicApp) is passed through to D365FO by a custom URL suffix. You need two minor modifications to make this work:

A JavaScript enabled button in D365S (in the previous example the “Create quote” button). The JavaScript code is hosted as a Web resource which holds the function to construct the full D365FO URL for a given button click (see simple example below).
X++ code in D365FO to interpret the custom URL part: form event handler extension on the form Initialize event which does the following (in this example):
- Reading the “customerid=” section in the URL
- Finding the associated customer record in the CustTable
- Setting the active customer in the Customer service form to this customer

D365S Web resource with JScript:

D365FO X++ code:

Due to the Azure AD enabled Single-Sign-On for D365S and D365FO, the D365FO client will open in about 2 seconds after clicking the JavaScript enabled button (the first time is always a bit slower due to caching). And given the Master-Slave 1-click approach which is similar to intra-D365FO behavior, most users will perceive this as ‘seamless’.

Evaluation of the concepts

I prefer the Master-Slave approach over the Hub-2-Hub approach for the following reasons:

With the Hub-2-Hub approach, it is not possible to pass parameter values which concern new records. A good example here is the D365S to D365FO new quotation flow (see Master-Slave picture above). In this context, it is critical to pass the D365S Opportunity ID to D365FO and store it on the new Quotation record created in D365FO. Only then this new Quotation record can, after making it a D365S record through CDS synch, be linked to the original Opportunity record in D365S (which prevents breaking integrity in D365S).
The Master-Slave approach emphasizes a true distinction in ownership of the Customer entity – This provides clarity to the users that they should always start their process in the D365S 360 degree Customer dashboard. By offering direct access from D365S to all selling features in D365FO, access to the customer related forms can also be minimized and/or reduced to view-only. This simplifies the use of both systems for average users.
The Master-Slave concept saves the user clicks compares to the Hub-2-Hub approach.

The strongest point of the Hub-2-Hub approach is that it minimizes customization. But I think looking at the minimal development effort required to implement the Master-Slave concept, this does not outweigh the negative arguments listed above.

Evaluation of the overall solution

Although this solution was well received by the customer in this particular project context, there are some critical notes to make:

The solution presented in this blog post relies on the fact that the D365FO Kernel allows inclusion of custom URL parameters. Since this may change in the future (malicious code may also be triggered in this way!), this is a true weak spot.
Ideally the base part of the URL is not hard coded in JScript but is retrieved dynamically based on an ‘Active environment’ parameter set in D365S.
It could be considered to make the parameter name (in the example above: “customerid”) configurable by a parameter. This could also avoid potential conflicts with new ‘Kernel’ parameters Microsoft might introduce. On the other hand, if the custom URL suffixes are regarded as a “service”, this would vote against this “configurable” approach.

I hope this inspires you in your creative thinking! At least it inspired me.. Special thanks to Stefan Ebert, Georg Braun and Markus Blässle for their creative input to make this happen!

Happy DAX’ing and happy New Year!

Patrick

1yr with AX7 Data Management – My top 12 learnings

Tue, 11 Oct 2016 00:00:00 GMT

We’re so used to the speed of current evolvements around Microsoft Dynamics AX (or should I say Dynamics 365 ;-)), that most of us who are interested in integrations have familiarized themselves for long with concepts like data entities and entity stores. In this blog post, I would like to go beyond the ‘obvious’ by sharing 1 year of personal project experience in working with AX7 Data Management.

To enable you to quickly scan this blog post according to your own points of interest, I have listed my experiences in a top 10-alike pattern in random order.

One year with AX7 Data Management: My top 12 learnings

How to be really sure that code changes have been effectuated for data entities
I’ve seen a lot of situations where a data entity, after updating the code for an AX7 data entity, showed unexpected behavior. In most cases, the people releasing the software had not forgotten to click the Refresh data entities button in the AX7 Data Management’s Framework parameters after release. As it turned out, this is not always sufficient. I apply the following procedure which never failed me so far:
Delete the mappings on data projects which build upon the data entities to be refreshed (this is required for step 2).
• Note: in a typical project, multiple data projects may exist building upon a single data entity. This table browser query will help you find the respective data projects quickly: https:///Default.htm?mi=SysTableBrowser&prt=initial&cmp=&tablename=DMFDefinitionGroupEntity&limitednav=true
Delete all the data entities from the Data entities list which are software updated.
Now refresh data entities through System administration > Workspaces > Data management > Framework parameters > Refresh data entities. The deleted entities will re-appear.
Re-create the mappings w you deleted earlier.

2. Public cloud versus OneBox behavior
Most DEV environments and even TEST environments will be based on OneBox installations. This is either a local VM with all AX7 components running on premise through Azure authentication or similar VM hosted on Azure. The differences in behavior between these SQL Server OneBox environments and Azure SQL powered Public Cloud environments are decreasing by the minute due to bug fixes and general advancements. But, be careful here. Do not expect that your Public Cloud environment will 1:1 behave like your TEST environment did before. Be on the safe side and reserve some time and budget for regression testing when taking the step from TST to UAT environment. I’ve seen SSIS errors on Public Cloud environments which never occurred in a TEST environment.

Custom interface tables based on event triggers as an alternative for outbound business logic
It seems obvious to leverage the out-of-the-box SQL change tracking capabilities when only updated or new records are to be shipped. It also seems obvious to implement business logic at interface run time. But do not make these decisions too quickly. Weigh all the pros and cons of the different options as there are more options with the framework than you think. For example, in my project I faced a scenario in which it was a requirement to ship out all unique AX7 barcodes to a legacy system. So all new unique barcodes were to be gathered from a multi-company situation (barcodes are company-specific) and additional business logic was needed to gather additional information. I could have leveraged SQL change tracking and implement outbound business logic. But that would have made the interface fairly slow. Instead, I implemented business logic on event basis: whenever a barcode was inserted, business logic evaluated the cross-company uniqueness of the barcode, gathered the additional meta data and inserted this data into a custom interface table when evaluated positively. The result: fast interface export.
Management-by-exception and 1-click troubleshooting
While building experience with AX7 Data Management, my customer quickly came up with feedback that they expected a dashboard showing all exceptions and 1-click troubleshooting capabilities. So I analyzed their requirements versus the current AX7 Data Management capabilities by creating a diagram highlighting the steps required to troubleshoot the most common scenarios:

My main conclusions:
• It requires a minimum of 3 clicks to retrieve the cause of an error.
• There’s no form in the AX7 Data Management area which allows for creating personalized tiles to show interface exceptions.
• The Recent runs tabbed list on the IT data management workspace does not provide a complete overview of meta data – Some critical information is missing, for example:

The list does not show the number of records in the source file, so it is difficult to identify source to staging errors.
The list only shows the status for the data job run as a whole, but often a job run concerns multiple data entities, which all have their own individual status.

• It is not possible to retrieve the original file name in case a file has been uploaded through the Data Management API. Therefore, it is not possible to trace an (erroneous) data job run back to the original uploaded file.

So I decided to design a new parent-child inquiry form which shows all possible meta data for all interface runs:

As it is an inquiry form, the user has the ability to create a personalized filter on a single interface, group of interfaces in any status or group of statuses. As per standard AX7 functionality, the filter can be turned into a tile which can be placed on any workspace and pinned to the main dashboard:

So in case an exception occurs, a tile highlighting interface exceptions will show value 1 or higher after refresh. This is the trigger for a system administrator to take action. A click on the tile will directly show the exception in the interface form. By the meta data in the form the system administrator will have a good sense about a possible cause, so he can perform a direct click to the relevant error log (execution log or staging table log). So it will only require 2 clicks to inquire the cause of any error. The challenge here then is to get the number of exceptions down to 0 again after fixing the issue and re-running the respective data job.. ;-).

Staging table errors in outbound interfaces
Some of you might question why it would be needed to set Skip staging to false on a data project – In other words: to utilize the staging table in an outbound interface process. Well.. it is a requirement if you need to run business logic in your outbound interface.
Although you need to be careful with the implementation of business logic in outbound interfaces, as this can significantly degcrease the performance of your outbound interface, there are scenarios in which you really need it. For those scenarios you need to be aware that outbound staging to target processes handle errors in a different manner than inbound staging to target processes:

Inbound processes perform validations against standard or custom AX7 business logic. In case of errors, the respective staging record goes into error status and import for the particular record is withheld.
In outbound processes, custom business logic may put the record into error status, but the export is NOT withheld. AX7 is designed to output the records AS-IS, independent from the status of the record in the staging table.

So if you want to have your outbound staging to target process behave in a similar fashion as your inbound equivalents, here’s my recommendation:

Implement business logic to validate your data for output as you’d do in an inbound process. So promote your staging records to the status as required.
Implement a routine to delete all the staging table records which are in error state. This will ensure only records which pass your validations are outputted (if you want to allow any output in case of errors).

As a result of this deletion, errors are thrown at Execution log level instead of staging table level:

Leveraging the power of XSLT transformations
In many outbound interface scenarios, AX7 will have to output columns in a certain order or output specific column names. This can all be handled by using sample files in your data project. But if you need more complex transformations, AX7 offers the ability to leverage XSLT transformations. And these can be really powerful. Some examples:

Transform root node and main child node names to reflect the required XML schema. In this example the root node “Document” is transformed to “ItemCatalog” and the child node “”> is transformed to “InventTable”.
Sort all XML elements and values in the output XML alphabetically:

Minimize non-auto mappings on a data project
When I architecting the various interfaces in my project, I was very keen on including value conversions, default values and other transformations in my data project mappings to provide some flexibility for future adjustments. However, I started to lose enthusiasm when I found out that default values and conversions were not allowed for enums. I even decided to completely push all possible transformations to third-party systems when I found out that mappings cannot be migrated at all. Background here is that with every update on a data entity, you might have to do a lot of manual re-mapping across different environments (DEV, TST, UAT) – see learning 1 above.
Interesting in this context is that if you open the manifest.xml file from a zipped data package, you hit upon the placeholders for the mappings and transformations. But they’re simply not populated as per the current functionality:

Therefore, I’d advice (for now) to reduce the use of sample files (non-auto column mappings), default values, transformations and value conversions to a minimum.
Be careful with LOAD PROJECT!
In most demos I have seen concerning AX7 Data Management, interface imports and exports are always run through the data project’s Load Project feature (System administration > Workspaces -> Select data project > Load project). But what they never tell you in those demos is that this feature can completely screw up your interface mapping without you being aware of it. Some examples:
• In case of a component entity, using the Load Project feature will have your child entity mapping removed from the mapping if the child entity is not in the upload file.
• If you deleted an auto mapped column from the interface mapping on purpose, the deleted mapping will have been re-created automatically if you upload the file through the Load Project feature.
• The Load Project feature will cause a mapping for a particular field to be removed if the field was originally mapped but no longer exists in the source file uploaded.

Hence my recommendation for running interfaces manually is as follow:
• Only allow system administrators who know what they’re doing to run interfaces manually through the LOAD PROJECT feature.
• Align the third-party software version, samples files used for unit/functional testing and AX7 data entity software version. This will reduce the risk on issues in the Load Project scenario:

• Teach all other users to run interfaces manually through either of the following methods which will never affect the interface mapping:
a. The Run Project feature.
b. A setup which allows users to drop files in folders which are uploaded and processed through the Data Management API.

Export data entity field characteristics
In any integration scenario it is critical to have clear agreement about the interface mapping including the responsibility for data transformations. I always use the following structure for architecting my integrations:
In order to quickly populate the template with the main characteristics of all the data fields in the respective AX7 data entity (field length, field type etc.) I use the following table browser URL: /Default.htm?mi=SysTableBrowser&prt=initial&cmp=&tablename=DMFDefinitionGroupEntityXMLFields&limitednav=true
Deprecation of multiple value separator
In previous versions of AX, a ‘multi-value separator’ (for example “,”) could be defined to allow multiple values in 1 column to be imported/exported. In AX7 this is no longer needed due to ‘normalization’ of the list of data entities. A typical example of the classical multi-value import was for example the store/address book association. In AX7, this association is created through a separate data entity “Retail store address book”:
Load monitoring
By enabling the processing monitoring job on the Recurring data job..

..the Queue trend is populated:
This enables you to inquire the load and adjust the frequency of the processing job – In case of high Peaks, ensure to run the interface more often to reduce the load per job run:

12. Staging table fields versus data source fields
While architecting interfaces it is important to be aware of the relevance of the difference in scope of data between a data entity and its staging table. For example, you may choose to include additional tables and fields for filtering on the data to be exported, but you may not want to make this data available for export.

AX7 – My top 12 highlights

Sat, 27 Feb 2016 00:00:00 GMT

I onboarded the AX7 technical conference this week with the impression that AX7 was ‘just’ a new UI exposing AX2012R3CU9 code on a new platform with some adjustments to ‘make it work’ on the new Azure platform. But already after the first keynote I felt embarrassed about my initial thoughts which quickly made place for respect and amaze for this enormous leap in technology.

I can only compare the way Dynamics AX has developed in the last 10 years with a boxing game: initially, the boxers tease and challenge each other by some quick targeted moves. But the game really takes off when one of them finds his ‘punch’ (say with the release of AX2012) The opponent is driven towards the corner of the boxing arena and receives punch after punch, each punch having bigger impact. I think AX7 is at the core of this ‘momentum’. In this blog I’ll share the main ‘punches’ and my personal interpretation of what’s going on. If you want to dig deeper then visit the publicly available AX7 wiki.

AX7: my top 12 highlights

1. AX7 is a true Saas/Paas platform leveraging the Azure platform natively

In the literal words of Mike Ehrenberg, AX7 is not a ‘lift-and-shift’ of AX2012R3CU9: simply shifting the on premise platform to the cloud, as many competitors have done. Instead, Microsoft managed to separate the AX7 application layer and platform layer which allows AX7 to run on native Azure technology. So the many future Azure enhancements ahead will directly impact AX7 positively. With SQL Server, this was an exciting journey as only from CTP6 Microsoft was able to run SQL Server natively on an Azure VM.

2. AX7 is Microsoft-wide achievement

When Steve Jobs and Bill Gates ignited Silicon Valley with their innovations, they could achieve a lot with a very small team of people. Now with AX7, the AX7 ‘pyramid’ is constructed from many already giant pyramids, ranging from SQL Server technology to Azure to BI technology. Each ‘pyramid’ has its own big team of experts. It’s amazing how Microsoft has managed to blend all these broad specialisms into the AX7 product.

3. With the AX7 release is not incorporating new technology but driving new technology

With AX7 being a cloud-first platform fully in line with Microsoft CEO Satya Nadella’s focus for service orientation, The new Dynamics is now fully in the spotlights within Microsoft. Beyond that, AX7 is on a path where it drives the other Microsoft teams to enhance, instead of ‘just’ incorporating existing technologies in new releases. A good example is real time analytics. With the market demand for real-time analytics, AX had to move away from non-real time data warehousing and SSAS technologies in favor of the new read-only secondary database (replicated in seconds) and AX entity database (replicated in minutes). Instead, AX required new technology to allow true real-time analytics. This technology was found in new SQL Server 2016 in-memory technology, leveraging indexing on columns instead of the conventional row-based indexing.

4. Enriching AX functionality exactly according to market demand

AX7 has been developed with more customer, partner and ISV involvement than ever. But that’s something that could have been done 10 years ago. The new thing here is that telemetry data is collected from actual AX7 usage. Based on already available data up to the recent RTW release, Microsoft has been able to identify exactly which areas of AX are used most and which areas have been customized most. This enables targeting all available development capacity spot on. Can you imagine what will happen if AX7 is enrolled to thousands of customers worldwide?

5. The visible part of the iceberg is getting smaller

With the release of AX7, the production environment will no longer be accessible by customers or partners. Microsoft ‘frees’ the customers and partners from the responsibility to maintain this environment, to apply patches and – again in the words of Mike Ehrenberg – to allow the partners and customers to focus more on unleashing the potential of the software for their respective business instead of being bothered by ‘low-value’ activities. I prefer to use the metaphor of an iceberg in the water to express what is going on: the part of the iceberg which is really visible and tangible (the part above the water) gets smaller and smaller and the part under water bigger and bigger. In other words: less effort and ‘hassle’ against maximum value and service.

6. AX7 resolves many implementation pain points

With AX7 Microsoft truly expresses an ongoing focus for shortening the AX implementation cycle. Many pain points are now solved by Microsoft:

Fast code deployment: the application code base has now been split up in multiple models. Code can now be compiled and deployed in smaller chunks and in a much faster way.
Easier code upgrades: the conventional ‘layering’ option is still there, but custom code can now be implemented as an extension (applicable to not all but many AOT elements). As such, the extension is completely segregated from the out-of-the-box code base, which makes customizations much more flexible, especially regarding future upgrade scenarios.
Shipping of configuration data among environments: with so called data packages (a collection of data entities which are an aggregated collection of tables and views), sets of data can easily be shipped across different environments. Optionally the data in the data package can be edited in Excel prior to deployment as the data package is physically nothing more than a zipped collection of Excel files added up with a header and manifest XML. So called ‘process data packages’ can even associate a data package with a specific process in the LCS process library. You can have a new sales order posted in a brand new originally empty environment in minutes.
Multi-purpose task recorder: the new task recorder can record tasks in AX not only to train people on any task in the software (including interactive guided clicking), but can also be used to automate testing. The recorded tasks (including the data captured) can be made part of new software builds to automatically regression test the build before deployment.
Retail: activation of a cloud-POS or mPOS device can now be done through an easy-to-use wizard. As there’s only 1 channel database for all your channels in the new cloud topology, the wizard preopulates the channel database URL and also the stores the device can be associated for (based on the Azure AAD user > worker > store address book > store association).

7. Life Cycle Services (LCS) is no longer a recommendation but a requirement

In many ways adoption of life cycle services can no longer be avoided. Some scenarios to highlight this:

As AX7 no longer provides access to production environments, the telemetry LCS offers is crucial in identifying issues. Then consecutively, LCS is leveraged to quickly deploy a representative environment, load relevant reference test data and simulate the issue with a task recording.
Data packages and other assets are all stored in the LCS Asset Library. LCS is the central repository for storage and deployment across the various environments. LCS has a hierarchical structure: a ‘corporate’ section which allows re-using ‘assets’ among different projects.

8. Support 2.0

AX7 re-defines the way you can support your users. Having your users record the issue they face through the task recorder is one thing. Being able to create a ticket from AX7 which is directly converted into a Visual Studio work item for the support engineer is another thing. But the earth shaking thing here is the available telemetry for the support engineer: as AX7 continuously logs system performance and system activity along the way, the work item is automatically enriched with a ‘snapshot’ of the state of the system at the time the issue occurred: what browser did the user utilize, what batch jobs were running, what was the user doing exactly etc.

9. Business process orientation

Although I think the business process repository on LCS could be organized and incorporated in a better way, the AX7 UI offers a brilliant new way its application functionality is exposed to the business users: workspaces. Conceptually the workspaces have many similarities with the good old role centers: tiles are the new cues, tabbed lists are the old enterprise portal enabled listpages (although the new ‘tab’ structure is very handy) and charts and links can be embedded as role center offered (although far more sophisticated now). However, a user can now be offered a numerous number of workspaces, all supporting a specific set of tasks related to a process the user is affiliated with. The set of workspaces together forms the user’s dashboard.

10. New features

OK, they might have become a bit less prominent because of the other hot topics mentioned, but AX7 still offers some great new features – a glimpse of the less prominent features:

Finance: cross company general journal entry: creating and posting journals without having to switch companies.
Retail: being able to download updates to mPOS from the device form directly (self-service installer). Process: a new update is made available by IT and a store manager can download and run the update package locally in the store.
Enterprise search and ribbon search: just type ALT+G to open enterprise search and then type “U C”. AX7 will come up with “Unit Conversion” immediately – one click and you’re in the form. You can even navigate to other pages by simply changing the URL. Type ALT+Q and you can search for a specific function in the ribbon, regardless of the tab group the function belongs to.

11. The cloud: a new way of thinking

With the actual AX7 cloud release you have to be aware that basic on-premise functions have suddenly become a challenge. Here’s a shortlist of challenges and the way Microsoft managed to tackle those:

Accessing on-premise files and folders is not possible from the cloud: AX can no longer poll on-premise folders and pull-in files for processing, for example in recurring integration scenarios. Instead, files have to be pushed into the cloud. Here’s how: Microsoft has exposed an API which allows a source system to push a file into Azure blob storage accompanied by an enqueuing message which flags a recurring AX batch job that there’s a file to be processed. Microsoft will ship an application to do the enqueuing (and similar dequeuing for exports) on behalf of the source (or target) application. This application will be able to work with processing folders (IN, ERROR, COMPLETED etc.) as many customers were used to with the DIXF batch job (see this blog post by Kurt Hatlevik).
Printing on local network printers is not possible from the cloud. Again, we need an on-premise pull mechanism here. For this case Microsoft will ship a ‘document routing agent’ which will query an Azure queue for messages which contain meta data for new documents to be printed (stored in Azure blob storage). Network printers can be configured on a specific form in AX7 which will allow the Azure queue message to contain meta data which printer the ‘document routing agent should send the document to.
Hot keys: the different browsers have different reserved hot keys. To stay as much in line as possible with current hot keys users are familiar with, Microsoft had to be creative. For example, the ‘new record’ hot key CTRL+N has now been replaced by ALT+N, which is still pretty familiar for hot key addicts.

12. Relatively low threshold for adopting AX7

It’s a challenge to develop and release a new product which involves so many paradigm shifts. But it’s pretty handsome if you manage to make the adoption of the new product still relatively easy – very important when it comes to minimizing costs for existing partners to make the shift and minimizing costs for customers to upgrade. Why I do consider the AX7 adoption to be relatively easy:

Developers will have to embrace Visual Studio, the AOT is re-arranged a bit and developers have to get used to work with the extensions and file based code and packages. But X++ is rather untouched (only enriched at some points) and for those who have some Visual Studio experience, a lot of ‘new’ things is actually common to .NET/Visual Studio in general.
Yes, the cloud environment requires a new way of thinking as stated in the latter point, but many things will be managed by Microsoft ‘under water’. As always, understanding the concept is much easier than having to become a specialist in an area.
With the earlier CTPs operating the new AX7 UI was a bit different than the familiar AX2012 navigation. But with RTW the similarities are very clear again. The area pages are back in town for those who worship them, form dynalinking (selecting a different record in the listpage is reflected in a details form) is working flawlessly and the good old ribbon and favorites are still there.

I hope this post inspired you to embrace the new AX7 features without any fear to make the jump.

Happy renewed DAX’ing.

Patrick

Dynamics AX Retail: how to automate processes and interfaces in a combined way

Wed, 23 Dec 2015 00:00:00 GMT

Business processes and interfaces often go hand in hand: it’s often either a business event triggering an interface or a business event following up on incoming interface data. For example: automatically creating pick orders and updating the related order statuses in the e-commerce frontend. In our aim for maximum comfort for both the business user as well as the system administrator, how can we automate this type of business events and interfaces in a combined way?

In this blog post I’ll boil the necessary standard AX ingredients into a configuration which allows you to reach this goal. I’ll also show you how to leverage this configuration to completely automate the following example flows:

Flow 1: importing transactions from a (third-party) sales channel, converting the transactions to sales orders and sending an acknowledgement to the channel that the order import was successful or unsuccessful.
Flow 2: updating the on hand stock for the different channels and sending the stock levels to the sales channels.
Flow 3: creating picking lists for the incoming orders and informing the sales channel on the ‘picking’ status.
When you have this running, it’s like simulating a factory with Lego: when your order pickers come in in the morning they’ll find the picking lists on the printer; the sales channels can work with the latest stock levels and order statuses and by leveraging role center your system administrator sits in his control room waiting for any exception to occur which requires intervention – Sounds almost like a real factory… right?

Example scenario

Below you find an overview of the example scenario for this blog post with the 3 flows mentioned – I highlighted the manual and automated tasks in blue/red respectively and painted processes and interfaces as solid lines/dotted lines respectively.

In this particular example, the (third-party) sales channel has some specific requirements:

It requires to receive an acknowledgement if sales order creation in AX was successful or not.
It requires to receive stock updates at least every 5 minutes.
It requires to receive information that the picking process for the items has started. This information is then available in the customer’s ‘My account’ section on the online store and is sent to the customer by SMS.

AX ingredients to automate

The ones of you with AX/Retail background will probably recognize how the different tasks in the example scenario can be supported by standard AX/Retail batch jobs:

The interface flows 1A, 1C, 2B and 3B can be supported by Retail Distribution Schedules (Retail > Periodic > Data distribution > Distribution schedule), although they might have to be custom configured. The distribution schedules are to be configured as batch jobs.
The transaction to sales order conversion can be supported by the Synchronize online orders batch job for e-commerce (Retail > Periodic > Synchronize online orders) or the Post statement batch job (Retail > Periodic > POS posting > Post statement) for POS. In this case we work with the Synchronize online orders batch job as in the example scenario we have to pick order placed at the online store.
Creating pick orders can be automated in multiple ways but in this case we leverage the Picking workbench (Inventory management > Periodic > Picking workbench > Picking workbench).

Important to notice here on the side is that both the Synchronize online orders batch job as well as the Picking workbench will try to reserve items automatically, independent from the settings in the Accounts Receivable parameters (> General > Sales > Reservation) which might be on Manual. For this reason, and for the reason that picking will be tied to one specific point of time (for example 7AM), as opposed to the other batch tasks which will run periodically throughout the day, the 3 different ‘flows’ (combinations of business event and interface) are to be scheduled in their own frequency. This also provides the opportunity for an order manager to allocate additional on hand stock to fulfill open order requirements which failed to claim stock at import before the picking workbench runs automatically.

So… from helicopter perspective AX has a pretty good coverage, right? But we need to bundle the automated tasks in 3 groups which can be scheduled in their own frequency, instead of having to manage 7 batch jobs which are to be aligned in timing individually. How do we organize that?

Towards a solution

From the requirements stated above we can distill that we need the following configuration:

2 batch jobs for flow 1 and 2, to be scheduled periodically with high frequency (for example every 5 minutes)
1 batch job for flow 3, to be scheduled periodically with low frequency (once or twice a day at fixed points of time)
Multiple batch tasks per batch job, to be run sequentially.
Custom distribution schedules to support our interface flows, to be run as batch tasks. Assumption: we leverage standard AX/Retail CDX for interfacing, so we ship data in/out through the AX channel database for the sales channel.

Point 4 can be covered by defining new or updated Distribution schedules supported by new or updated scheduler job definitions. As an example for element 3A (Send pick status/information) of the example scenario above:

Points 1 and 2 are covered by the standard AX batch framework (System administration > Inquiries > Batch jobs > Batch jobs) which offers the ability to define standard recurrence patterns. Example batch job for flow 3:The difficulty in is point 3. For being able to schedule the batch jobs as separate batch tasks under the umbrella of one batch job, three things need to be arranged:

Customization part 1: batch class adjustment. The respective business logic is to be enabled for running as a batch task. In AX 2012 R3 there are two main ways of developing batch classes:
- Through extending the RunBaseBatch framework: the ‘classical’ framework.
- Through extending the SysOperation framework: the framework introduced with AX 2012 which is recommended by Microsoft.
  All current Retail batch jobs (my reference version: AX2012R3CU8) (still) rely on the RunBaseBatch framework. In this framework, batch classes can be enabled for running as batch tasks by setting the CanGoBatchJournal method to return TRUE – Here’s an example for the RetailCDXScheduleRunner which contains the business logic for running the Distribution schedules in batch:

Note: if this method is not added or returns FALSE, the batch class is simply not selectable as batch task under a batch job.
Note 2: you may have noticed that for some Retail batch jobs, Microsoft has added the CanGoBatchJournal method in AX 2012 R3.

Customization part 2: passing parameters to the batch task. As a batch class is sometimes built upon an existing form which is designed to pass parameters to the batch class (in the example above: a parameter which indicates which distribution schedule to run), it is not always possible to set the right parameters in the context of the batch job configuration form. See below for an example of this issue: the Distribution schedule form (Retail > Periodic > Data distribution > Distribution schedule) passes the selected distribution schedule as a parameter to the batch form when configuring a batch job. In the Batch tasks form (System administration > Inquiries > Batch jobs > View tasks; see figure below), this parameter cannot be set (only consulted). So in areas where the batch class parameterization relies on existing forms, some controls and business logic have to be added to enable the batch class to receive the right parameter values.

Additional configuration. In our example scenario as described above, we have to enforce the tasks to run sequentially. But how? Standard AX provides some tools here:
- Batch task conditions. On the batch task we can configure AX to run task 2 after successfully finishing task 1. Here’s an example for our flow 3 above:

Server configuration. When applying batch task conditions you’ll notice that some batch tasks (for example a Retail Distribution schedule) create subtasks which cannot be conditioned. So if Task 1 creates a subtask 1a, then task 2 could already start running in a separate (parallel) thread. Therefore, to force AX to run the main tasks sequentially, the number of threads on the batch Server configuration (System administration > Setup > System > Server configuration should be put down to 1 – Note that Maximum batch threads indicates the number of tasks AX can potentially execute in parallel per batch job.

Note: the batch tasks which are to be enforced to run sequentially, need to be allocated to the batch server with Maximum batch threads = 1 setting specifically:

Customization part 3. Even with the configuration under point 2 we are still not fully sure tasks will run sequentially. Some tasks are running in CIL (Common Intermediate Language) and cannot be checked for completion by AX; AX considers the tasks to be done when handed over to CIL and will use the thread for the next task. In other words: neither batch task conditions nor server configuration will work. In the example of task 1 being executed in CIL, the only thing I could think of was to implement a pause to make sure task 1 is finished before task 2 takes off. I’ve built this ‘pause’ in the form of a custom batch enabled class which allows me to set a number of seconds of pause before continuing to the next task – it’s not ideal, but with some testing it does its job perfectly. Here’s the code in the run method of the RunBaseBatch based BatchPause class (parameter: secondsPause):

public void run()
{
int i;
;
try
{
i=sleep(secondsPause*1000);
}
catch(Exception::Deadlock)
{
retry;
}
catch(Exception::UpdateConflict)
{
throw Exception::UpdateConflict;
}
catch
{
throw Exception::Error;
}
}

This business logic can then be used in the form of an additional batch task to pause the batch job for the number of seconds specified; here’s is an example for flow 1:

The cherry on the cake

Configuring the batch framework as described above adds another level of abstraction to the already powerful Retail CDX management framework in the Retail module. But when you top it off with a dedicated role center configuration, you can make a serious step towards Management by exception across the board. To stay with the example above, at business side you could implement an ‘exception cue’ for all orders which could not be directly fulfilled at import (flow 1A):

At IT management side you could implement an ‘exception cue’ for all (Retail) batch jobs errors. Clicking this cue will allow you to quickly descend into more details and solve the errors quickly and effectively:
I hope this has given you an impression on how you can create sets of interrelated automated tasks, whether having a business or integration background, to simplify the management of your complete solution and maximize comfort for your business and IT users.

Happy DAX’ing and happy holidays!

Patrick

Financial dimensions on web orders vs non-Retail orders

Tue, 27 Oct 2015 00:00:00 GMT

Say we have non-Retail B2B sales and B2C weborders in 1 company. We sell the same items through these channels. We expect the financial dimension values for the same financial dimensions to appear on our order lines but… they differ between these scenarios… HUH?And at invoicing we get stuck (posting is cancelled) as in the B2C scenario we miss a mandatory financial dimension value CostCenter… ARGGGGG!

Cause

For the cause of this issue we have to dig into the way the financial dimensions are inherited in each scenario. Here’s what happens in both scenarios:

In both scenarios the sales order header inherits its financial dimension values from a main entity: in the Retail web order scenario that’s the Retail channel, whereas the customer is the main entity in the non-Retail sales scenario (criterion for ‘being’ a non-Retail sales order: RetailSalesTable.RetailOrder == NoYes::No). This difference makes sense as in the Retail scenario you might want to keep track of the sales channel your actual sale originated from.

But there’s one major difference between the scenarios when it comes to sales order header to sales order line inheritance: in the non-Retail case, the financial dimension values on the sales order header and item level are both (AND) inherited to the sales order line. In the Retail web order case it’s a case of OR: if the item contains default dimension values, then these are copied to the sales order level, otherwise the sales order line inherits the sales order header dimension values.

Solution

For non-Retail sales, the inheritance happens as follows:

\Classes\SalesLineType\InitFromSalesTable (inherit from SO header to SO line):
\Classes\SalesLineType\InitFromInventTable (inherit from item to SO line):
So, the actual magic is in the salesLine.mergeDimension method. Here’s where the AND card is played.

In my customer’s scenario it was important to align the two cases so the Account structures setup in General ledger could be consistent in terms of mandatory financial dimensions. Therefore I decided to apply the merge to the Retail web order area as well – It also makes the situation more consistent in general. For the Retail web orders the financial dimension values are set as part of the Retail > Periodic > Synchronize online orders batch job. The applied business logic resides in the RetailOrderManager::createEachOrderLine method. So in order to align the cases I only needed to call the Merge functionality in this context as well. A change of only 2 lines of code:

One last addition to this: if you look at how financial dimensions are set on customer orders (created from Retail POS through Real Time Service) and sales orders which are created through Retail POS statement posting, then we see another variant (RetailOrderManager::mergeOrderDefaultDimension):
In this context, financial dimensions are set on sales order header level as a hierarchical merge of even 4 dimensions: customer, Retail store, Retail terminal and worker. You’d better be aware of all these variants when setting up AX financially ;-).

Happy DAX’ing!

Patrick

Dynamics AX Retail third-party POS and e-commerce integration – Control dashboard

Sun, 18 Oct 2015 00:00:00 GMT

In my main blog post regarding Dynamics AX Retail/third-party integration I mentioned leveraging Real Time Service to allow middleware (like Microsoft BizTalk) and third-party systems to report to AX on the results of their actions. As the meta data is written into the standard Dynamics AX Retail download sessions and upload sessions forms, these forms will now serve system administrators to have visibility and control even beyond their the Dynamics AX Retail CDX landscape (CDX -> see rectangle in the middle, beyond CDX -> bottom rectangle):

In this blog I’ll provide the X++ source code to enable this and I’ll mention some tiny features I added to the download sessions and upload sessions forms to maximize control on our third-party integration. First I’ll introduce you to the topic by explaining a bit more on how to extend Real Time Service for access from third-party systems.

How to extend Real Time Service

What I like most about Real Time Service is that it provides a single-point-of-access to any method in AX from the outside world. With ‘single-point-of-access’ I mean that with Real Time Service we don’t need to expose an endless number of services if for example we want to get customer information, create a customer and cancel an order – we can stick to 1 service only as the AX method we want to call into through the service is a parameter on the service. So if we want to create a customer in AX we tell the service to invoke the RetailTransactionService::newCustomer method and we provide the service with the input parameters the respective method in AX expects:

Message for calling the Real Time Service (newCustomer method) = The “sender”AX HQ newCustomer method = The “receiver”

So basically you can call into any method in AX from third-party systems just by passing the right parameters and sending the information in the right message format. In a later blog I’ll provide more details in this particular area – for now I stick to mentioning some basic ‘rules of play’:

Real Time Service is designed to call into the RetailTransactionService and RetailTransactionServiceEx classes in AX by default. Although even this is configurable in the web.config file of the RealTimeService application on IIS, I’d stick to this standard. In practice this will never block you from leveraging any method in AX. Just follow the path Microsoft is on since the AX 2012 R3 release, to leverage the methods in the RetailTransactionService and RetailTransactionServiceEx classes only as ‘gateway’ to other classes not including any business logic in itself. It basically comes down to the following structure (see left hand side):
Before going into customization, carefully check the richness of the current RetailTransactionService class first. Microsoft has exposed more than 100 Retail related methods packed with business logic through this class. So the most common Retail operations (which are normally exposed to the native AX channels ;-)) are available. In general, it’s easy to expose existing business logic by just creating a custom ‘gateway’ method to call into existing business logic. So rule of thumb should be that minimal coding is required, otherwise you might have chosen the wrong path.
Put all custom (‘gateway’) methods in the RetailTransactionServiceEx classes. Invoking these methods is nothing different than invoking the methods in the RetailTransactionService class except for replacing the node InvokeMethod by InvokeExtensionMethod in the example XML above.

Custom RetailTransactionServiceEx:: newRetailCDXMetaDataRecord

With this custom method (see source code below), both Retail > Inquiries > Commerce data exchange > Upload sessions and Download sessions table/forms are served. This is directed by the downloadMessage boolean field which is 1 (true) if the respective interface flow is a download flow (AX -> native AX/third-party) and 0 (false) if the respective interface flow is an upload flow (naïve AX/third-party). The artifacts prefixed with ‘pmo’ are discussed in the next section of this blog. First I’ll list you the input/output variables for the method – the array element type indicates what the type of parameter is that your XML message to Real Time Service requires for the variable. Note for the X++ code that I included many validations to ensure errors are not raised due to XML variable type to AX variable type conversion issues.

///

/// Method is added by PMOU (16.06.2015)
/// Method is used for creating a RetailCDXDataStoreSession record which reflects CDX meta data
/// BizTalk leverages the method to report on the result of its part of the AX/third-party channel interface chain
/// In doing so, system administrators have complete visibility and control across the complete interface chain including integration to third-party channels
///

///
/// Container with entity keys
///

public
static
container newRetailCDXMetaDataRecord(boolean downloadMessage,
RetailCDXSharePath         sDataFileOutputPath,
RetailCDXDataGroupRefRecId     sDatagroupRecId,
RetailConnJobId         sJobId,
RetailCDXRowsAffected         sRowsAffected,
RetailCDXScheduleRefRecId     sScheduleRecId,
RetailCDXSessionNumber     sSessionNumber,
RetailCDXDownloadSessionStatus     sDownloadStatus,
pmoRetailCDXLinkedSessionRefRecId     sdsLinkedSessionRecId,
RetailCDXDataStoreRefRecId     sdsDataStoreRecId,
str         sdsDateApplied,
str         sdsDateDownloaded,
str         sdsDateRequested,
RetailCDXMonDataSyncMessage     sdsMessage,
RetailCDXDownloadSessionStatus     sdsDownloadStatus,
RetailCDXCheckSum     uCheckSum,
RetailCDXDataStoreRefRecId     uDataStoreRecId,
str         uDateCreated,
str         uDateUploaded,
RetailCDXFileSize         uFileSize,
RetailCDXHqUploadSessionID     uHQUploadSessionId,
RetailConnJobId         uJobId,
RetailCDXMonDataSyncMessage     uMessage,
RetailCDXUploadSessionRerun     uRerun,
RetailCDXSessionNumber     uRerunForSessionId,
RetailCDXRowsAffected     uRowsAffected,
RetailCDXScheduleRefRecId     uScheduleRecId,
RetailCDXUploadSessionStatus     uUploadStatus,
RetailCDXTryCount         uTryCount,
RetailCDXPacketFilePath     uUploadPath,
RetailCDXSessionNumber     uUploadSessionId,
pmoRetailCDXLinkedSessionRefRecId     uLinkedSessionRecId,
pmoRetailCDXDataSource     uDataSource)
{
boolean         success = false;
str             error = ”;
Counter         infologline = infolog.num();
int             fromLine;
str             xmlResult = ”;
container         tmpResult;
RetailCDXDownloadSessionRefRecId     downloadSessionRecId;
RetailCDXDownloadSession     retailCDXDownloadSession;
retailCDXDownloadSessionDataStore     retailCDXDownloadSessionDataStore;
RetailCDXUploadSession     retailCDXUploadSession;
RetailCDXUploadSessionLog     retailCDXUploadSessionLog;
RefRecId         sessionRecId;
RetailCDXDateApplied         sdsdateAppliedUTC; //UTCDateTime RetailCDXDateDownloaded     sdsdateDownloadedUTC; //UTCDateTime RetailCDXDateRequested     sdsdateRequestedUTC; //UTCDateTime RetailCDXDateCreated         udateCreatedUTC; //UTCDateTime RetailCDXDateUploaded     udateUploadedUTC; //UTCDateTime
try {
fromLine = Global::infologLine();
// Validate if this is an AX inbound or outbound flow
if (downloadMessage == true)
{
// If SessionNumber is 0 then create a new session
if (ssessionNumber == 0)
{
ssessionNumber = RetailCDXSessionIDGenerator::getNextSession();
}
if (ssessionNumber > 0 && sjobId && sdatagroupRecId && sdsdataStoreRecId)
{
// Populate table RetailCDXDownloadSession
ttsBegin;
retailCDXDownloadSession.clear();
retailCDXDownloadSession.initValue();
if (sDataFileOutputPath)
{
retailCDXDownloadSession.DataFileOutputPath = sDataFileOutputPath;
}
retailCDXDownloadSession.DataGroup = sdatagroupRecId;
retailCDXDownloadSession.JobID = sjobId;
if (srowsAffected > 0)
{
retailCDXDownloadSession.RowsAffected = srowsaffected;
}
if (sscheduleRecId)
{
retailCDXDownloadSession.Schedule = sscheduleRecId;
}
retailCDXDownloadSession.Session = ssessionNumber;
if (sdownloadStatus)
{
retailCDXDownloadSession.Status = sdownloadStatus;
}
retailCDXDownloadSession.insert();
sessionRecId = retailCDXDownloadSession.RecId;
ttsCommit;
// Populate table retailCDXDownloadSessionDataStore
ttsBegin;
retailCDXDownloadSessionDataStore.clear();
retailCDXDownloadSessionDataStore.initValue();
if (sdslinkedSessionRecId)
{
retailCDXDownloadSessionDataStore.pmoLinkedSession = sdslinkedSessionRecId;
}
retailCDXDownloadSessionDataStore.DataStore = sdsdataStoreRecId;
if (sdsdateApplied)
{
sdsdateAppliedUTC = str2datetime(sdsdateApplied,123);
retailCDXDownloadSessionDataStore.DateApplied = sdsdateAppliedUTC;
}
retailCDXDownloadSessionDataStore.Session = sessionRecId;
if (sdsdateDownloaded)
{
sdsdateDownloadedUTC = str2datetime(sdsdateDownloaded,123);
retailCDXDownloadSessionDataStore.DateDownloaded = sdsdateDownloadedUTC;
}
if (sdsdateRequested)
{
sdsdateRequestedUTC = str2datetime(sdsdateRequested,123);
retailCDXDownloadSessionDataStore.dateRequested = sdsdateRequestedUTC;
}
if (sdsmessage)
{
retailCDXDownloadSessionDataStore.Message = sdsmessage;
}
if (sdsdownloadStatus)
{
retailCDXDownloadSessionDataStore.Status = sdsdownloadStatus;
}
if (uDataSource)
{
retailCDXDownloadSessionDataStore.pmoDataSource = uDataSource;
}
retailCDXDownloadSessionDataStore.insert();
ttsCommit;
success = true;
}
else {
error = ‘@PMO17’;
success = false;
}
}
else {
// If SessionNumber is 0 then create a new session
if (uUploadSessionId == 0)
{
// The upload session is synched from AsyncServerHeadOffice\dbo.UPLOADSESSION, so a new session ID should not interfere with the SQL ID range uUploadSessionId = RetailCDXUploadSession::getNextBizTalkSession();
}
if (uUploadSessionId > 0 && uJobId && uDataStoreRecId)
{
// Populate table RetailCDXUploadSession
ttsBegin;
retailCDXUploadSession.clear();
retailCDXUploadSession.initValue();
if (uUploadPath)
{
retailCDXUploadSession.UploadPath = uUploadPath;
}
retailCDXUploadSession.DataStore = uDataStoreRecId;
retailCDXUploadSession.JobId = uJobId;
if (uRowsAffected > 0)
{
retailCDXUploadSession.RowsAffected = uRowsAffected;
}
if (uScheduleRecId)
{
retailCDXUploadSession.Schedule = uScheduleRecId;
}
retailCDXUploadSession.UploadSessionId = uUploadSessionId;
if (uUploadStatus)
{
retailCDXUploadSession.Status = uUploadStatus;
}
if (uCheckSum)
{
retailCDXUploadSession.CheckSum = uCheckSum;
}
if (uDateCreated)
{
uDateCreatedUTC = str2datetime(uDateCreated,123);
retailCDXUploadSession.DateCreated = uDateCreatedUTC;
}
if (uDateUploaded)
{
uDateUploadedUTC = str2datetime(uDateUploaded,123);
retailCDXUploadSession.DateUploaded = uDateUploadedUTC;
}
if (uFileSize)
{
retailCDXUploadSession.FileSize = uFileSize;
}
if (uHQUploadSessionId)
{
retailCDXUploadSession.HqUploadSessionId = uHQUploadSessionId;
}
if (uMessage)
{
retailCDXUploadSession.Message = uMessage;
}
if (uTryCount > 0)
{
retailCDXUploadSession.TryCount= uTryCount;
}
if (uRerun > 0)
{
retailCDXUploadSession.Rerun= uRerun;
}
if (uRerunForSessionId)
{
retailCDXUploadSession.RerunFor = uRerunForSessionId;
}
if (uDataSource)
{
retailCDXUploadSession.pmoDataSource = uDataSource;
}
retailCDXUploadSession.insert();
ttsCommit;
// Populate table RetailCDXUploadSessionLog
ttsBegin;
retailCDXUploadSessionLog.clear();
retailCDXUploadSessionLog.initValue();
retailCDXUploadSessionLog.DataStore = uDataStoreRecId;
if (uDateCreated)
{
uDateCreatedUTC = str2datetime(uDateCreated,123);
retailCDXUploadSessionLog.DateCreated = uDateCreatedUTC;
}
if (uMessage)
{
retailCDXUploadSessionLog.Message = uMessage;
}
if (uUploadStatus)
{
retailCDXUploadSessionLog.Status = uUploadStatus;
}
retailCDXUploadSessionLog.UploadSessionId = uUploadSessionId;
retailCDXUploadSessionLog.insert();
ttsCommit;
success = true;
}
else {
error = ‘@PMO17’;
success = false;
}
}
}
catch(Exception::Error)
{
ttsabort;
error = RetailTransactionService::getInfologMessages(fromLine);
RetailTracer::Error(‘RetailTransactionServiceEx’,funcName(),error);
success = false;
}
tmpResult = [success, error];
return tmpResult;
}

Enhancements to the Download session and Upload session form

In the Upload session form, the Message field is not visible by default. As this field will contain any positive or negative feedback from our middleware or third-party system, this is the first thing to change:

As shown in the picture, in both Upload and Download session form I’ve added the Origin field which indicates which link of the interface chain is reporting to us (see second column). Last but not least I’ve added a Linked download session field to the Download session form which shows which session (read: record in the Download session form) the current record followed up on. In this way, you can easy filter the form to have all the meta data as reported by the different chains of the complete interface flow in one overview.
I hope this has shown you how powerful Dynamics AX for Retail can also be leveraged for third-party sales channel integrations.

Happy DAX’ing!
Patrick

Dynamics AX Retail third-party POS and e-commerce integration – Solution Design

Sun, 18 Oct 2015 00:00:00 GMT

So here’s the challenge… we have Retail CDX up-and-running… how do we hook up any middleware to integrate with third-party POS systems, e-commerce and other sales channels?

Hmmm… let me add one nuance here which we might overlook: how do we hook up middleware without touching the integrity of Retail CDX?

In order to answer this question I think it’s good to shortly refresh our minds about Retail CDX. Then we’ll take a look at my solution for both the batch processing as well as transactional types of interfacing.

Standard AX/Retail CDX

Dynamics AX Retail CDX builds upon the principle that data should efficiently be distributed to decentral sales systems like POS systems. This efficiency should maximize data actuality in the sales system, even in low bandwidth environments and even when large datasets (for example products) are exchanged. This principle is relevant as a sales system should have the highest possible uptime and therefore should never be directly dependent on interfaces with central systems for its day-to-day operations. Further, Retail CDX is designed for enterprise environments where multiple sales systems are to be served from 1 or multiple AX instances with a variety of time zones, currencies, local tax requirements etc. Keep this in mind when looking at CDX overview picture below which only shows a 1 AX instance/1 sales channel situation.

So what exactly is the flow of things within the CDX landscape?
For the batch processing side of the landscape, it’s basically all about the AX database, channel database and two ‘helpers’ to ship the data over between the two databases: a central helper (Async server; HQ side) and a local helper (Async client; channel side). The helpers only do something when they receive instructions. After receiving instructions, they do their job and write a standardized report on their job (meta data) back into their own meta databases. In addition, the local helper also reports his meta data to the central server helper, so all local meta data about everything happening in the CDX landscape can be synched back into AX by the Retail > Periodic > Data distribution > Process status messages batch job.

To be more precise:

AX hands over an instruction to download information to the central helper.
The central helper contacts the AOS which is the gateway to the AX database , downloads the required data, serializes that data into XML files, wraps these XML files into 1 binary file and outputs that file to a working folder which was configured in AX and synched to the central helper’s interface meta database as part of the CDX setup.
The local helper periodically requests the central helper if there any new instructions to be executed. The frequency for this request is defined in the Async server profile configuration in AX. In case of any data to be shipped, the central helper responds with a download instruction including a reference to the working folders. The local helper then writes this instruction into its interface meta database, executes the instruction (downloads the binary file), unpackages the binary file into XML files, de-serializes the XML files, injects the content of the files into the channel database, writes meta data about this operation into the interface meta database and reports the same information to the central helper (which the central helper writes into its own interface meta database).

Hooking up middleware: batch processing (non-real time)

So the actual transport of data between AX and the channel databases is done by XML files?
YES – let’s take a short look into those files (note: you can use the tool DDPACKVIEW for this) – the example here is the 1000 scheduler job for currencies:
As you can see the binary ‘package’ contains 1 header XML file and multiple XML files with data. The header XML files indicates which XML files containing data (‘data files’) are attached in the binary package and which operations should take place for the different data files (delete or insert/update). The data files ‘just’ contains the data to be deleted or inserted/updated:

As you can conclude from the “ax.” prefix of the tables, the files and data 1:1 reflect the target Retail channel database schema and data in AX, which is recognizable from the RecId value for the currency table. This is because we picked the AX 2012 R3 Retail channel schema for this example, where channel database schema and AX database schema are perfectly matched.

So why wouldn’t we just skip the channel database in the architecture and let middleware intercept and process the binary file with XML files?
I didn’t choose that path for the following reasons:

The AX/channel database integration is proven technology which goes well back into the history of the LS Retail solution (say 15 years ago). So ‘decoupling’ the binary file transfer from this architecture will impose project risks.
Some scheduler jobs (like the 1040 – product) ship over 80 XML files. Processing these quantities of XML files for 1 flow will impact performance negatively. Middleware specialists rather integrate through SQL server than through XML files as SQL server is faster and provides more technical opportunities for integration.
By using the CDX XML files within the middleware orchestration, we have to make the middleware aware of the AX data model in some way, while the AX channel database offers many out-of-the-box views and stored procedures which can be leveraged as an access gateway for the middleware solution.

Details of the solution for batch processing

So instead, I chose to hook up BizTalk to the channel database. But how could I trigger BizTalk to perform action without impacting the standard AX/Retail CDX integrity? To answer this question we need to cover the same dimensions which are covered by the standard AX/Retail Scheduler: WHEN, WHAT, WHERE. The WHEN refers to the trigger for the middleware solution to take action. Let’s take the example of a BizTalk application which is functionally linked to the 1000 scheduler job run in AX and technically to the local client’s helper for a specific channel. I took the following conditions as criteria for BizTalk to start taking action:

[Async client meta database].dbo.DownloadSession.JobId = [Job ID BizTalk application is designed for]
[Async client meta database].dbo.DownloadSession.Status = ‘4’ (=Applied)

In other words: the BizTalk applications is continuously monitoring the local helper’s interface meta database for in-scope channel database changes which have been confirmed by the local helper.

But at this point in time the BizTalk application does not have a clue yet about the WHAT yet: what data has been changed exactly? In order to answer this question I decided to imitate the behavior of the trigger to ship data on HQ side: SQL change tracking. See my other blog post for more details if you’re interested in this feature. Bottom line, a similar SQL change tracking mechanism has been be applied to the SQL database. BizTalk builds upon stored procedures which confront data from one more standard AX channel database tables/views or custom views with change tracking. Here’s an example for a stock availability flow:

The core table which triggers BizTalk to create an output file for the third-party system is the RETAILINVENTAVAILBILITY table (see directly behind the CHANGETABLE statement in the picture above). But the trigger can easily be on more tables and/or individual columns. Be aware that this is dependent on the requirements of the target third-party system.

So our situation is now as depicted below: a BizTalk application is monitoring the local helper’s interface meta database to spot for in-scope data changes. If this is the case, then the application fires a stored procedure to the channel database to check if the changes are relevant for the target system.

So.. say we have a situation where our BizTalk application has identified relevant data changes.. But how does the application know which system to deliver this data to:

The customer where I implemented this solution had over 20 e-commerce instances across the globe and different types of topologies varying from a central BizTalk instance/regional AX instance situation to a regional BizTalk/local AX instance situation. In that situation the WHERE question goes beyond the target system only. It’s about HOW to implement flexibility across the whole architecture as a whole. What if the customer decides to move a channel database to another server or if the central/regional topology is changed into a regional/regional topology? We cannot simply hard code all the server names, database instance names and service addresses into BizTalk. That would mean a redeployment of many applications if something is changed in the infrastructure landscape.

Analyzing these needs and looking over the powerful tools we have in hand with Retail CDX I came to an out-of-the-box solution. I decided to define BizTalk as a Retail store in AX and create a custom scheduler job 9000 (by configuration ;-)) to ship all technical/infrastructure related channel meta data to a special AXBizTalkConfiguration database, which is technically nothing more than the channel database for the virtual BizTalk channel. I only needed to add fields to AX reflecting the location of the third-party system, Async client database and meta database instance name:

Then in the AXBizTalkConfiguration a special view returns all the characteristics from the channels which would have had to be hard coded otherwise:

The information from the view is consumed by the BizTalk application at run time and utilized until the application is restarted. This prevents hard coding and having to republish BizTalk applications over and over again in case of changes. See my blog post Agility (blog post will follow) for more details if you’re interested in this feature.

So.. We’ve now made BizTalk aware of WHERE to deliver the output information in an agile way. Here’s the full solution overview including triggers:

Before we go along the triggers in the landscape, there’s one feature left unspoken. As we’ve seen for CDX, all local helpers indirectly report the results of their actions back into AX. This provides us the means to control the CDX landscape from our one and only AX headquarters.
When it comes to integration, everybody who has some practical experience in this area knows the by far biggest vexation in this area: FINGER POINTING :(. This is often a clear indication that either frameworks for efficient maintenance are missing or clear procedures for problem solving or the worst of all… both. So the best way to avoid this is to hook up both BizTalk and the third-party systems to existing CDX meta data framework – Let all systems in the landscape report the results of their interface related actions back to AX. In order to offer a standardized entry point for this I’ve leveraged Real Time Service to directly write into the standard AX/Retail Inquiries > Commerce data exchange > Upload sessions/download sessions forms. The result looks like this:

We can follow the chain of interface links up till its destination and easily intervene in case of any issues! See my blog post Control dashboard for more details if you’re interested in this feature. So let’s get back to our main solution above again and consider each and every trigger in the interface landscape:

Trigger 1: batch job in AX initiating the (potential) transfer of data to the channel database. Frequency: to be determined by the business.
Trigger 2: The local helper (Async client) requesting the central helper for new instructions. Frequency: as configured in the respective Async server profile in AX. Common setting: 1 minute or immediate.
Trigger 3: a BizTalk process polling for in-scope data changes. Frequency: immediate.
Trigger 4: a BizTalk process transforming source to target data and delivering this data. Frequency: immediate
Trigger 5: a BizTalk pocess reporting back to AX on the result of its actions. Frequency: immediate

So from this trigger landscape we can conclude that all links in the chain ‘follow’ the AX frequency: when the business decides to ship out products (1040 scheduler job) with a frequency of once an hour, the rest of the interface chain will follow… if the target system (the ‘unidentified’ trigger 6) supports this as well… For the upload flow (third-party to AX) this is no issue as the frequency of the AX pull job will still decide the ultimate frequency of data exchange. My advice: configure all the other interface links on ‘immediate’, so AX has full control over the frequency across the complete landscape.

Hooking up middleware: transactional information exchange (real time)

Most of you will be familiar with request/response mechanisms. But For the sake of completeness I’d like to spend a few more words on the transactional information exchange through Real Time Service as well. Most important to notice here is that I leveraged Real Time Service not only when timing or direct response was a requirement for an interface flow. Actually I weighed with every flow design where standard AX/Retail CDX could offer the best fit, independent from the type of interfacing on third-party system side. For example, I faced a situation where the third-party system sent customer data to BizTalk in a batch processing manner. But for my AX side it was better to leverage the standard AX/Retail RetailTransactionServiceCustomer::newCustomer method instead of trying to implement a new pull flow for batch processing customer records.

Further, I also applied the CDX meta data service call through Real Time Service (you know.. To report on the result of interface actions) to the real time flows. This gives a nice ‘ping-pong’ effect where the responses of 1 or multiple service calls are reported back to AX by another service call:

Conclusion

To conclude this overview of a Dynamics AX/Retail CDX based solution for third-party sales channel integration, I’d like to stress the comfort this solution brings to your day-to-day use of standard AX/Retail. Within AX you’ll be able to handle a third-party sales channel as a ‘normal’ AX Retail store or Online store. This means that also conceptually you can fully leverage standard AX/Retail to support your third-party sales channels:

You’ll even be able to create data groups which contain both native as well as third-party channels.

Some of you will argue here: yeah nice, but what about price calculations in the third-party channels or tax calculations?? For the native AX POS and online store, we have Retail server which manages to bring the same pricing logic to the local native AX sales channel as we have available on HQ – what about the third-party sales channels?? As I might spend another blog on this topic in the future, my answer for now is simple: either integrate your third-party channels also with Retail server or Real time service or limit your AX/Retail configuration options (for example for discount structures) based on what you third-party sales channels can support. In this latter case, my recommendation would always be to minimize the footprint of your sales channels and maximize the footprint of AX. It’s not for no reason that AX supports many localizations – you don’t want to re-implement all those localizations in your sales channels when rolling out your third-party system worldwide ;-).

Happy DAX’ing!
Patrick

Dynamics AX Retail third-party POS and e-commerce integration – Trigger mechanism

Sun, 18 Oct 2015 00:00:00 GMT

In my main blog post regarding Dynamics AX Retail third-party sales channel integration I presented the mechanism to trigger BizTalk to take action after an AX scheduler job run. In this blog I’ll provide some more details on this particular topic. Let me first explain how the change tracking mechanism is leveraged on AX HQ side and let’s then take a closer look on how change tracking can be leveraged on the channel database and Async client database for use by middleware like Microsoft BizTalk.

Change tracking on AX HQ

On AX HQ the Generate classes button on the Retail > Setup > Retail scheduler> Retail channel schema form runs business logic in the RetailCDXCodeGen class. One of the steps is here is to enable change tracking for all tables reflected by the subjobs in AX – this saves us the time of having to enable change tracking on SQL level itself (thanks MSFT ;-)). Another step is to create queries for each table which is defined as a subjob (Retail > Setup > Retail scheduler > Scheduler subjobs). Here’s what such a query looks like for the standard AX currency table (source: RetailCDXChangeTrackingSQL table):

As you can see in the CHANGETABLE and CHANGE_TRACKING_IS_COLUMN_IN_MASK statement, this query enables AX to list the records for which any of the indicated fields (columns) in the Currency table has been updated for a particular change tracking version (passed through with the %2 parameter). So when a scheduler job runs, AX does the following:

AX selects the set of tables which is relevant for the particular job run (here: the 1000 – Currency).
AX gathers the changes for these tables since the last change tracking version which has been addressed in the last job run. Note: this change tracking version is visible as ‘row version’ under Retail > Periodic > Data distribution > Distribution schedule > [Select schedule] > [History button in ribbon]:

Important here is that SQL Server does not keep change tracking history forever as this might expand the database size disproportionally in case of high-transactional use of AX. So to limit the number of transactions, SQL Server works with a retention period which can be set on the database properties in SQL Server:

The default retention period is 2 days. If the retention period is exceeded (read: if a particular job has not been running once in 2 days), AX has no other option than to run a full synch for the particular flow since the change tracking history is not completely available anymore. You’ll receive the following infolog in AX when running the scheduler job:

The change tracking information related to scheduler job and data group are only partially available, due to the AX database change tracking retention period setting. A complete snapshot of data will be generated instead.

Be careful with this… your target system might only be designed to receive changes and not the full load. What if your 1040 – Product flow would ship out its 150.000 items in the channel’s assortment… So be aware of this function when designing your specific interface flows.

Another point to consider is the ability to trigger a full sync manually. In AX 2012 R2 we had our A-jobs (forcing to synch changes only) and N-jobs (forcing a full sync). In R3 the N-jobs seem to have been removed from the scene, but that’s not really the case. In fact, a full data sync can still be triggered from the Retail > Setup > Channel integration > Channel database form:

So train your system administrators to handle these features in a way which suits your interface designs.

Change tracking on Async client and Channel database

As mentioned in my main blog post regarding Dynamics AX Retail third-party sales integration, the following conditions should be met before BizTalk actually starts doing something:

[Async client meta database].dbo.DownloadSession.JobId = [Job ID one of the BizTalk applications is designed for to process
[Async client meta database].dbo.DownloadSession.Status = ‘4’ (=Applied)

This might for example be the actual situation in the Async client database:

As each job ID will trigger a different output flow to the third-party system, the job IDs are coupled with a specialized BizTalk process each. However, to avoid having them all polling on the dbo.DownloadSession, a generic BizTalk process is taking care of that task. It’s this process that triggers another BizTalk process to handle the respective Job ID subsequently. In order for this generic BizTalk process to keep track of the records it has triggered specific BizTalk processes for, change tracking is enabled for the dbo.DownloadSession table. So after each query cycle, BizTalk writes the last change tracking version for the dbo.DownloadSession table into its own local administration. So at each cycle new records can easily be identified and validated. Filtering the records by both the criteria mentioned above and change tracking is done by stored procedure.

A similar mechanism has been applied to the channel database. Let’s take an example to illustrate that. Say we have two custom scheduler jobs in AX with job IDs 8000 and 8010 respectively. These jobs ship data for the following tables:

But only the following tables are supposed to eventually trigger output to be shipped to the third-party system:

Say that we update 1 record in table 1 within AX and 1 record in table 6 within AX. We run both 8000 and 8010 jobs for our channel, so we have CDX shipping the updates over to the related channel database and writing meta data records into the related Async client database. Consequently, middleware (in this example: BizTalk) kicks in to process the data to third-party. The full flow of things in this layer is pictured in the following overview:

BizTalk process #1 spots new finished job runs (status = 4) for both flows and triggers BizTalk processes #2 and #3 to figure out if there’s actual data to be shipped to the third-party system. Similar to the Async client change tracking mechanism, BizTalk processes #2 and #3 inquire the latest change tracking versions for their respective job runs. For both job runs this is #9000 in this example, but it could have easily been different for each of the flows.

So BizTalk process #2 runs a stored procedure to only have a recordset returned which addresses all changes later than version #9000 and changes to table 1 only. Example:

In this example, biztalk.[PMOCHANNELITEMSAVAILABILITVIEW] returns the dataset which is the baseline for the output to the third-party system. The standard AX/Retail table ax.RETAILINVENTAVAILABILITY is indicated as the CHANGETABLE (=table 1 in our example) and @LastSyncVer indicates the last change tracking version for this job run (version #9000 in our example). So if this stored procedure returns any data, then BizTalk process #2 will have work to do to transform and hand over the data to the third-party system. After successful processing, BizTalk process #2 registers a combination of Job ID and last processed change tracking version (#9001) in its administration to avoid any duplicate output in next processing cycles.

Process #3 has still the job ID 8010/change tracking version #9000 combination in his administration. So the same original change to table 1 can still trigger process #3 to process his output for the third-party system.

I hope this blog has given you a better insight of how you can trigger a middleware solution on top of standard AX/Retail CDX.

Happy DAX’ing!
Patrick

Dynamics AX Retail third-party POS and e-commerce integration – Introduction

Sun, 18 Oct 2015 00:00:00 GMT

With the ongoing hunt for implementing ‘omnichannel strategies’ by Retail companies and the booming Microsoft Dynamics AX market, there must be a huge demand for integrating Microsoft Dynamics AX Retail with third-party POS and third-party e-commerce platforms. But when I google the internet for keywords like “AX 2012”, “Integration”, “third-party”, “POS” and “e-commerce”, there are not too many hits. And if I find any hits, discussions quickly seem to head for the same solution: AIF (Application Integration Framework).

In my vision leveraging AIF in the Dynamics AX Retail area is a missed opportunity. To prove this and to provide more information regarding Dynamics AX Retail/third-party integration in general, I’ve decided to write a series of blogs regarding this topic. In this blog post you can find the solution which I’ve recently implemented for a globally operating Retailer based on: Standard AX RetailCDX (Commerce Data Exchange). Here’s a picture of the entire architecture as described in the blog.

And in these blog posts you’ll find more related details:

Dynamics AX Retail third-party integration – Trigger mechanism
Dynamics AX Retail third-party integration – Automate process AND data exchange
Dynamics AX Retail third-party integration – Control dashboard
Dynamics AX Retail third-party integration – Agility (blog post will follow later)

I hope this series of blogs will help to create more interaction on this topic online and inspire fellow DAX’ers around the globe to fully leverage the potential of Dynamics AX Retail. Last but not least I’d like to thank BizTalk specialist Simon Emmanuel Rivas for his valuable contributions to the design I’ll present in my blogs.

Happy DAX’ing!
Patrick

Dynamics AX methodology: complex integration designs – How to keep overview

Sun, 18 Oct 2015 00:00:00 GMT

Some people hate it.. some people cherish it as a means to eternalize their brilliant designs… DOCUMENTING. One way or the other, in the bigger projects you can easily fill up a local library with all the paper produced. That’s why a good solution design is crucial in these projects. If such a design covers 1 system like AX, it’s easier to come to a holistic view on the desired solution. But what if your solution spans across multiple systems, all requiring some installation, configuration and customization. How do you keep the overview without getting lost in the details? For this situation your solution design needs an extra helper: the UML sequence diagram:

In this blog I’ll publish you my personalized “UML Sequence Digram +”. It doesn’t ‘just’ show the sequence of flows as a UML Sequence Diagram typically does, but it highlights all of the following elements which together form your entire solution:

Installation elements (highlighted in brown)
Configuration elements (highlighted in red)
Customized elements (highlighted in purple)
Interface elements (highlighted in green)
Business tasks and automated process tasks (highlighted in blue)

Below you find two examples of the “UML Sequence Diagram +”, 1 for deploying the integration of a third-party online store with Dynamics AX 2012 R3 Retail and 1 for a specific interface flow of this integration (the item price flow).

Example 1: deployment of a third-party sales channel

Example 2: update price flow

The advantage of this diagram is that when you’re fully into the process of shaping and prototyping your solution, you can quickly keep your documentation in sync by only adding elements to the diagram with a specific ID and meta data. How? The UML Sequence Diagram is a Visio template (download link) which is baselined by a connected Excel template (download link). In this Excel template you create the elements and all meta data which you want to store for the elements. If you want to change the name of your element you can do it quickly in Excel – the Visio template is only for having the graphical overview of all the elements. I also have the habit of saving screenshots to a file location with the ID of the element and a suffix number (in case of multiple screenshots per element).
The templates are especially helpful in agile projects and environments where customer requirements tend to change on daily basis. You can postpone the moment on which you descend into fully detailed installation guides, setup guides etc. until you have confirmation from the customer about your design and/or prototype. And if you chose your level of meta data registration well enough, you have collected just enough details to provide very thorough documentation without the risk on leaving out any details.
All the related final documentation will follow the structure of the UML sequence diagram. Here’s an example for an installation guide (download link) which fits to example 1 above – Note the element IDs between parentheses:

This approach also saves people from having to dig through a linear document in order to find what they want to know. They can identify the specific element which they’re interested in and quickly descend into the right document and element section by the color of the element and the ID of the element.

I hope this will help you improving the overall quality of your documentation and managing your document libraries in general.

Happy DAX’ing!
Patrick