Restore attachments for testing

  • May 16, 2025

Background

Many of my clients rely on document attachments as an integral part of their business processes. Having these attachments available during testing is critical for validating related functionality. It’s often more efficient to use existing data for testing rather than recreate data and reattach documents manually.

Attachments are not restored with the database, as they are stored separately in Azure Blob Storage. To my knowledge, Microsoft does not provide an endorsed method for restoring attachments alongside a database. While base X++ code includes a feature to “heal” document attachments when links are broken due to a database restore, my testing shows this only repairs documents originally uploaded in the current (ie. Test) environment.

Edit May 19, 2025: Microsoft supports restoring attachments with your database if you use the “Refresh database” option. It is not supported with “Point-in-time restore Prod to Sandbox”. Note they will restore all attachments which may be an issue if they are using too much of your storage capacity.

Prior state

In the past, the Azure Blob connection string was accessible via custom code. With direct access, testing attachments was straightforward—you could either copy the entire blob to your test storage and run a SQL script to repair the connections, or call the production blob directly from your test environment’s X++ code, restoring thousands of documents in minutes.

These methods are now deprecated or no longer functional in many environments, and likely won’t work on any tenant by the end of this year. Microsoft has published documentation about these changes:

Solution

This solution supports testing document attachments under the new restrictions. This also works in older versions. I now prefer this method even when the old access is still available as it requires almost no manual effort once the code is in place.

The approach uses a “just-in-time” method: when the test environment attempts to retrieve an attachment that’s missing, the code calls your production F&O instance, downloads the attachment, stores it in your test blob, and then returns the file to the caller. From the user’s perspective, it behaves seamlessly—aside from a slight delay when the file is first retrieved. Once downloaded, the document is stored and accessed normally going forward.

Key considerations

Production Calls: This solution makes real-time calls to your production F&O instance. Avoid it if the number of calls could impact server performance. In my experience, most test scenarios require fewer than 10 documents, so the impact is minimal.

Deprecated Service: The code uses the base service DocumentHandling, which was deprecated on 1/6/2020. As of version 10.0.42 (2025), it still works. If Microsoft removes it, you could replace it with a custom service or another base class. While there are OData entities that return documents, I couldn’t find one that isn’t tied to another entity (e.g., sales orders). There are viable alternatives—this was simply the most straightforward one available today.

Storage bloat: Using this process, then restoring the database, and repeating, you may be orphaning attachments in your test Azure Storage Blob. Depending on the scope of your testing and your storage capacity, this may be a problem you want to address and is not covered in this article.

Prod code: This code is designed to have low impact if deployed to production, but there is no requirement to deploy it there. I haven’t deployed it to any production environments and therefore have not tested it in Production. However, I understand the preference for code parity across environments. Pay close attention to the logic in reAttachFile, it needs to be changed to match your production URL.

Use at your own risk after proper due diligence.

Solution setup

You’ll need to create a Client ID/secret to connect to Production. Then configure Entra ID (formerly Azure AD) in your Production environment to allow the connection—this is the same as any other D365 integration. Reference: D365 services integration

Before using the provided code, set up parameters for the following values (code for creating these parameters is not included). Refer to the OAuth2 documentation for further detail:

  • DocuResource – Your Production F&O URL
  • DocuTenantId – Your Azure AD tenant ID
  • DocuClientId – The client ID from above
  • DocuRestoreSecretPassword – The client secret value (not the secret ID)

Note: I recommend encrypting the client secret. If encrypted, the value is lost whenever the database is moved and must be reconfigured. See the SysEmailSMTPPassword table for an implementation example.

The code

Key points about the implementation:

  • The code extends DocumentViewer, enabling it to intercept attachment retrieval when users click the paperclip icon in the UI to manually view attachments.
  • It also extends DocuActionFile, which allows it to intercept file access during background or automated processes. If your environment has custom file retrieval methods, you will need to apply similar extensions.
  • The Microsoft JSON parser code I found limited file sizes to 4 MB. I extracted and modified the parsing logic to support significantly larger files (up to ~1.5 GB). For files beyond this size, a new parsing approach is recommended such as deserializing to data contract classes, but such large attachments may not be ideal for this architecture anyway.

Conclusion

After the initial code and setup, this is almost as “Set it and forget it” as you can get.  In subsequent database restores from prod to test, you will just need to reconfigure the 4 parameters.  Or just the password parameter if you deployed this to prod and performed the setup there.  Or no config if you didn’t encrypt the client secret (Not recommended).

Code

Extensions to base code

[ExtensionOf(classStr(DocuActionFile))]
final class sciDocuActionFile_Extension
{
    public System.IO.Stream getStream(DocuRef _ref)
    {
        //In the next call, so we must get ahead of the file error before the call, otherwise control moves to the outer catch statement
        //This code first checks if the file has been modified by NonProdRestore, then if the environment is Prod
        //So minimal processing time is lost for files already restored and/or if this code is added in prod
        sciDocumentAttachmentHelper::reAttachFile(_ref.docuValue());

        return next getStream(_ref);
    }
}

[ExtensionOf(classStr(DocumentViewer))]
final class sciDocumentViewer_Extension
{
    /// <summary>
    /// Refresh the attachment on the attachment viewer if restored to a test environment 
    /// </summary>
    public void ReferencedRecordChangeCommand()
    {
        DocuRef docuRef;
        if (docuRefRecord == null)
        {
            docuRef = formDataSource.cursor();
        }
        else
        {
            docuRef = docuRefRecord;
        }

        if(docuRef)
        {
            DocuValue docuValue = docuRef.docuValue();

            if(docuValue.ModifiedBy == 'NonProdRestore')
            {
                sciDocumentAttachmentHelper::reAttachFile(docuValue);
            }
        }

        next ReferencedRecordChangeCommand();
    }
}

Custom class to download file and apply

class sciDocumentAttachmentHelper
{
    public static void reAttachFile(DocuValue _docuValue)
    {
        if(_docuValue.ModifiedBy != 'NonProdRestore')
        {
            //Already fixed
            return;
        }

        try
        {
            boolean isProd = true;//Assume true to be safe
            var infrastructureConfig = Microsoft.Dynamics.ApplicationPlatform.Environment.EnvironmentFactory::GetApplicationEnvironment().Infrastructure;
            System.Uri hostUri = new System.Uri(infrastructureConfig.HostUrl);
            str hostUriFormat;
            if(hostUri)
            {
                hostUriFormat = hostUri.GetLeftPart(System.UriPartial::Authority).ToLowerInvariant();
            }

            if(hostUriFormat)
            {                                                                //IMPORTANT - update this to match your environment name (ie. prdCompany or CompanyProd, etc)
                isProd = strScan(hostUriFormat, 'PROD', 1, strLen(hostUriFormat));//If we have an environment uri which does not contain prod, it is non prod 
            }

            if(isProd)
            {
                return;//Do not run in prod
            }

            DocuRef docuRef;
            select firstonly docuRef
                where docuRef.ValueRecId == _docuValue.RecId;

            if(!docuRef)
            {
                return;
            }

            DocuType docType = docuRef.docuType();
            if(docType.FilePlace != DocuFilePlace::Archive)//Archive = "Azure storage"
            {
                return;
            }

            if(!sciParameters::find().DocuResource)
            {
                warning("Complete the setup in custom document parameters to retrieve document attachments for testing");
                return;
            }

            str base64File = sciDocumentAttachmentHelper::getFileStrFromProd(docuRef.RecId);
            if(!base64File)
            {
                return;
            }

            System.IO.Stream stream = Binary::constructFromContainer(BinData::loadFromBase64(base64File)).getMemoryStream();

            //See docuActionFile method createFileAttachment
            Microsoft.Dynamics.AX.Framework.FileManagement.IDocumentStorageProvider storageProvider = Docu::GetStorageProvider(docType, false, curUserId());

            if (storageProvider != null)
            {
                str uniqueFileName = storageProvider.GenerateUniqueName(_docuValue.OriginalFileName);
                str fileNameWithoutExtension = System.IO.Path::GetFileNameWithoutExtension(uniqueFileName);
                str fileExtension = Docu::GetFileExtension(uniqueFileName);
                str fileContentType = System.Web.MimeMapping::GetMimeMapping(_docuValue.OriginalFileName);

                _docuValue.reread();
                _docuValue.selectForUpdate(true);
                _docuValue.StorageProviderId = storageProvider.ProviderId;
                Microsoft.Dynamics.AX.Framework.FileManagement.DocumentLocation location = storageProvider.SaveFile(_docuValue.FileId, uniqueFileName, fileContentType, stream);

                if (location != null)
                {
                    if(location.NavigationUri)
                    {
                        _docuValue.Path = location.get_NavigationUri().ToString();
                    }

                    if(location.AccessUri)
                    {
                        _docuValue.AccessInformation = location.get_AccessUri().ToString();
                    }

                    ttsbegin;
                    _docuValue.update();
                    ttscommit;
                }
            }
        }
        catch
        {
            warning("File attachment is missing and error occurred restoring the file");
        }

        return;
    }

    public static str getFileStrFromProd(RefRecId _docuRefRecId)
    {
        str bearer = sciDocumentAttachmentHelper::getBearerToken();
       
        return sciDocumentAttachmentHelper::getDocument(bearer, _docuRefRecId);
    }

    public static str getDocument(str _bearer, RecId _docuRefRecId)
    {
        try
        {
            sciParameters sciParameters = sciParameters::find();

            str url = strFmt('%1/api/services/DocumentHandling/DocumentHandlingService/getFile', sciParameters.DocuResource);

            System.Net.Http.HttpClient httpClient = new System.Net.Http.HttpClient();

            System.Net.Http.HttpRequestMessage request = new System.Net.Http.HttpRequestMessage(System.Net.Http.HttpMethod::Post, url);

            System.Net.Http.Headers.AuthenticationHeaderValue authBearer = new System.Net.Http.Headers.AuthenticationHeaderValue('Bearer', _bearer);

            var headers = request.Headers;
            headers.Add('Authorization', strFmt('Bearer %1', _bearer));

            str payload = strFmt('{ "_docuRefRecId": %1 }', _docuRefRecId);

            var contentStr = new System.Net.Http.StringContent(payload, null, 'text/plain');

            request.Content = contentStr;

            var response = httpClient.SendAsync(request).Result;

            response.EnsureSuccessStatusCode();

            str result = response.Content.ReadAsStringAsync().Result;//"null" or json with "Attachment" is the base64 str

            if(result != 'null')
            {
                //Adapted from RetailCommonWebAPI::getMapFromJsonString(result); but increased the max json length (previously only supported 4mb files)
                System.Web.Script.Serialization.JavaScriptSerializer ser = new System.Web.Script.Serialization.JavaScriptSerializer();
                ser.MaxJsonLength = 2147483647;//Signed int32 max
                CLRObject dict = ser.DeserializeObject(result);
                CLRObject value = dict.get_Item('Attachment');
                str stringValue = value;

                return stringValue;
            }
        }
        catch
        {
            Error("Error downloading file from target environment");
        }

        return '';
    }

    public static str getBearerToken()
    {
        try
        {
            sciParameters sciParameters = sciParameters::find();

            str authUrl = strFmt('https://login.microsoftonline.com/%1/oauth2/token', sciParameters.DocuTenantId);
            str clientSecret = sciDocuRestoreSecretPassword::find().passwordEdit(false, '');

            var httpClient = new System.Net.Http.HttpClient();

            var request = new System.Net.Http.HttpRequestMessage(System.Net.Http.HttpMethod::Post, authUrl);

            var content = new System.Net.Http.MultipartFormDataContent();

            content.Add(new System.Net.Http.StringContent('client_credentials'), 'grant_type');
            content.Add(new System.Net.Http.StringContent(sciParameters.DocuClientId), 'client_id');
            content.Add(new System.Net.Http.StringContent(clientSecret), 'client_secret');
            content.Add(new System.Net.Http.StringContent(sciParameters.DocuResource), 'resource');

            request.Content = content;

            var response = httpClient.SendAsync(request).Result;

            response.EnsureSuccessStatusCode();

            str result = response.Content.ReadAsStringAsync().Result;

            Map map = RetailCommonWebAPI::getMapFromJsonString(result);

            str bearer = map.lookup('access_token');

            return bearer;
        }
        catch
        {
            Error("Error authenticating to FO production to restore file");
        }

        return '';
    }
}

Questions?

Contact Us

Reach out with any feedback

Always happy to answer questions

Built by Mike Schiermeyer with Wordpress, StartAce templates, ChatGPT, and Gemini