Releases: awslabs/visual-asset-management-system
v2.3.1
[2.3.1 - Hotfix] (2025-11-21)
Bug Fixes
- CLI Fixed bugs with sentinel object check, file upload exception returns, and pattern updates
Chores
- Web Added checks to web yarn install custom installers to look at which viewers are enabled/disabled before installing the dynamic libraries. This was mostly to reduce install and deployment times to not include viewer assets that are not enabled for the end-user.
- Web Updated to disable licensed file viewers by default in their configuration file
- CLI Updated CLI to require python 3.12 minimum and updated dependency versions (Click to 8.3.1 for Sentinel object changes for default parameters)
- Updated documentation
v2.3.0
Major Change Summary:
• New VAMS CLI Tool - Complete command-line interface with robust file handling for large-scale automation and integration workflows
• Overhauled Search & Asset Management - Redesigned asset and file search system with enhanced UI, advanced filtering, and improved location services integration
• Advanced File Visualization System - New plugin-based viewer architecture with new CesiumJS, BabylonJS, PlayCanvas, VNTANA, PDF, Video, and Text viewers plus modal popup access
• Enhanced Pipeline System - Auto-deployment registration capabilities, new CAD/Mesh extraction pipeline, Gaussian Splat toolkit, and streamlined backend dependencies
• Improved Asset Links & Metadata - Extended support for 4x4 Matrix, WXYZ, JSON, GEOJSON, GEOPOINT types with multiple parent/child relationships
• Performance & Security Improvements - Enhanced API Gateway authorizers with IP restrictions, asynchronous large file upload processing, and restored VPC lambda support
• AI-Assisted Development - Integrated CLINE and Kiro workflow rules for AI-powered coding assistance and improved developer experience
⚠ BREAKING CHANGES
All APIGateway authorizers were swapped for custom lambda authorizers to provide more flexibility in implementing additional functionality. This may cause issues with your organization so please review with your security teams. Authorizer changes may require forced cache resets on API gateways if new authorizations are not following new rules set. (https://docs.aws.amazon.com/cli/latest/reference/apigatewayv2/reset-authorizers-cache.html)
Changes to BatchFargate CDK construct naming for use-case pipeline naming may require you to deploy CDK without batch pipelines and then again with to properly re-deploy them. Not doing this with existing deployed pipelines (Metadata 3D Labeling and PcPotree) will result in a CDK deployment error within ECS Fargate. This may also require you to update your VAMS pipeline/workflow lambda function names after re-deployment.
In order to get lambdas to work behind a VPC again (broken as of V2.2), MFA for roles cannot be supported if Cognito is on and all lambdas are behind a VPC (CDK config flag) or OpenSearch provisioned is turned on (CDK config flag).
OpenSearch has new indexes and requires the data migration script or new re-indexing tool script to be run on existing assets and files to re-index open search with existing data.
Recommended Upgrade Path: Run upgrade script for the new OpenSearch indexes which will re-index content infra\deploymentDataMigration\v2.2_to_v2.3\upgrade
Features
- CLI VAMS now has a CLI tool that can be used to automate VAMS operations. It includes operations so far for authentication, database, asset, assetLinks, assetLinkMetadata, metadata, metadataSchema, tags, TagTypes, search, featureSwitch, and files. More operations to match API functionality to come in future releases such as more admin functionalities of VAMS.
- CLI has logic for asset uploading and downloading and optimized for many and large files
- CLI contains some experimental industry commands to help with automation of processing PLMXML files and doing asset-tree GLB combining
- New asset export API
/database/{databaseId}/assets/{assetId}/export POSTto make it easier for downstream tool integration to have a single call to fetch all information about an asset, all its related data, and asset link sub-tree information (including auto-fetching pre-signed URLs). Integrated into CLI to support easy fetching and file download logic. - Web The website viewer system has been rewritten to support a plugin-based dynamically loaded viewing system which allows for much easier capability to add new viewers and adds more functionality. Documentation can be found at:
web\src\visualizerPlugin\README.md- Support for multiple viewers per file types which is now controlled with a drop-down as part of the viewer
- Support to define which viewers are for multiple files or single files
- Support for custom parameters as part of viewer plugin configuration which allows for token configuration for paid/ISV integrations
- Support for custom code, UI, and dependency management for each viewer. Also supports lazy loading of plugins when needed for a viewer.
- Viewer is now shown both on the View File page and as a modal pop-up from the file manager for easier quick access
- Added a PDF viewer for
.pdfextension - Added a text viewer for
.txt,.json,.xml,.html,.htm,.yaml,.yml,.toml,ipynb, and.iniextensions - Added the CesiumJS viewer for
.jsontileset files which can load subsequent other files referenced in the asset (even if not selected for viewing directly). This is an initial/basic CesiumJS viewer implementation with default options as part of this release. Note: RequiresallowUnsafeEvalFeaturesCDKconfig.jsonconfiguration flag to be turned on (off by default). - Added BabylonJS-based Gaussian Splat viewer for
.plyand.spzsplat files - Added PlayCanvas-based Gaussian Splat viewer for
.plyand.sogsplat files - 3D Online viewer now has additional UI added to support basic extra functionality
- 3D Online Viewer once again will also support
.plyfile extensions for viewing (previously switched to PotreeViewer only) - Added the VNTANA 3D Model licened viewer to the viewer plugin system for
glbfiles. Head to VNTANA.com for license purchasing and then enable this viewer inweb\src\visualizerPlugin\config\viewerConfig.json.
- Overhauled the file and asset OpenSearch system, APIs, indexing, and user interfaces
- Assets and files are now split into two separate OpenSearch indexes; the old index will remain and will not be deleted for auditing and/or migration purposes; this causes breaking changes that require a migration script to re-index all assets/files for search
- Asset link relationship data will now additionally be indexed (excluding asset link metadata for now)
- UI Assets (now "Assets and Files") has a completely new search page with many new filtering capabilities and options.
- Web Search map view will now allow for many more metadata fields to be used for adding map marker or area placement (any asset with
location(GP/GS) andlongitude(string or number) /latitude(string or number) combination metadata will show up) - Search now has its original API of
/searchand a new/search/simpleAPI for a simplified search input - Implemented a new CDK config option in
config.app.openSearch.reindexOnCdkDeploythat can trigger a complete index clear and re-index of assets and files. This can also be used as CDK context argumentreindexOnCdkDeployfor the cdk deploy command. Note: Only use this after having CDK deploying at least once with v2.3 changes, otherwise the reindex may not work or error. - A new CDK custom tool section and migration scripts have been added to help manually trigger a reindex outside of a CDK deploy
- Maps on the backend and UI frontend are updated to use the new location service APIKey method and removes the older raster map and place functionality
- Note: This removes the last place that cognito identities are used which means the location services functionality can now be used for external IDP solutions. Cognito is no longer required to enable location services. Only requirement now is commercial cloud partition (GovCloud doesn't support APIKey implementation).
- Note: This change removes the cognito authenticatedRole and association with the identity pool. Unauthenticated role (no permissions assigned) still remains for now as it is needed for basic auth login by the web Amplify-SDK v1.
- Web Added a draggable splitter in ViewAsset page between the file manager tree view and details panel
- Added a new API endpoint for asset file streaming (similar to asset preview auxiliary files) at
GET /database/{databaseId}/assets/{assetId}/download/stream/{proxy+} - Added .clineRules and .kiro for AI workflows for AI-assisted development for VAMS backend API development, CDK development, and CLI development
- All HTTP APIGateway authorizers were swapped for custom lambda authorizers.
- New Lambda Layer specifically with libraries for the lambda authorizers
- New support for CDK configured IP range restrictions for API Gateway calls that are managed in the authorizer
- Added new uploadFile backend logic with an SQS queue to handle final processing of large >1GB files asynchronously. This prevented APIGateway->Lambda timeouts (30 seconds)
- Added WXYZ, Boolean, Date, 4x4 Matrix, Geoshape, GeoPoint, LLA (Latitude Longitude, Altitude), and JSON asset link metadata value types.
- Web Added
Matrixstatic asset link type metadata fields with relevent field types. - Web Defaulted
rotationstatic asset link metadata field to WXYZ field type (from XYZ)
- Web Added
- Asset link parent-child relationships now support an additional key of
assetLinkAliasIdthat can be added to allow multiple parent->child relationships of the same assets. This is common in scene or engineering assembly build-outs where a parent may contain multiple of the same type of asset below it (i.e. same screws on a panel or same trees in a forest scene). - Web Changed Pipeline Edit/Create to make Asset Type and Output Type a required string text field. This removes the last place that requires specific VAMS extensions to be preloaded. These fields usages are expected to be overhauled along with overall pipelines in a future release.
- Refactored createWorkflow to not require the stepfunctions library anymore which entirely removes the additional heavyweight lambda layer created specifically for this function. ...
v2.2.0
[2.2.0] (2025-09-11)
HIGHLIGHTS
- Complete overhaul of asset management and file handling with robust asset and file level versioning, asset multi-file support, multi-bucket support, and enhanced UI management
- New file operations including archive, delete, move, copy, download/sharing, and attribute viewing capabilities
- Advanced viewer support for multi-file selection and new file types including video and audio
- Enhanced asset and file thumbnail preview support
- Improved file upload/download with pre-signed URLs and multi-part support
- Improved file relationship and relationship metadata support
- Support for global databases for pipelines and workflows
- External OAuth2 IDP authentication configuration option
⚠ BREAKING CHANGES
- CDK Configuration files must be updated to include the new required fields. See ConfigurationGuide.md and template configuration files for new formats.
- Asset and Database DynamoDB table fields and formats have changed, which require using the migration scripts after CDK deployment to update the new field values. See /infra/deploymentDataMigration/v2.1_to_v2.2/upgrade/v2.2_to_v2.3_migration_README.md for details on using the migration scripts to upgrade your DynamoDB databases after deployment.
- Due to VPC subnet breakout changes, this may break existing deployments. It is recommended to use an A/B deployment if you run into subnet configuration issues.
- Due to Cognito changes, a new Cognito user pool may be generated on stack deployment. To migrate existing users from the previous user pool, follow the following blog instructions: https://aws.amazon.com/blogs/security/approaches-for-migrating-users-to-amazon-cognito-user-pools/
Recommended Upgrade Path: A/B Stack Deployment with data migration using staging bucket configuration and upgrade migration scripts for DynamoDB tables in ./infra/upgradeMigrationScripts
Contributions
- Lockheed Martin Corporation (LMCO) - LMCO has significantly contributed to this release with both external and internal pull requests (#204)
Features
- Database, Pipeline, Workflow, Tag, Tag Types, Role, and Constraints id/names no longer need to follow as strict regex guidelines. New Regex: ^[-_a-zA-Z0-9]{3,63}$
- AssetId no longer needs to follow as strict regex guidelines. New Regex (regular filename regex): ^(?!.[<>:"/\|?])(?!.*[.\s]$)[\w\s.,'-]{1,254}[^.\s]$'
- File paths no longer need to follow as strict regex guidelines and now allow for deep pathing. Some restrictions apply to specific input paths for auxiliary asset previews and pipeline output paths.
- The asset upload API and backend along with many associated supporting asset API backends have been rewritten to support new features, security, and performance improvements.
- The old uploadAsset, uploadAssetWorkflow, and s3scoped access APIs and backend have been removed
- A new uploadFile (initialize, complete, createFolder), createAsset, and assetService (edit asset) have been created to support separation of assets and files. UploadFile now fully supports S3 Signed URL uploads for better security and performance (replaces providing UI with scoped S3 access).
- ScopedS3Access removal provides benefits as previous implementations had issues with scoped role timeouts, different authentication implementations in VAMS, parallelization issues, which prevented file validation, asset file overwrite issues, and more.
- New AssetUploads DynamoDB created to track uploads between initializations and completions
- IngestAsset API, intended for backend data system ingresses, wraps the new APIs as an all-in-one API caller.
- UploadFile is now split into two stages for upload, which allow for multiple files and multiple parts per file to be specified for better performant uploads of large files
- Assets now are better built to support a range of different files, including no files. The separation allows for better reliance on S3 functionalities to support file versioning.
- AssetType on assets are now specified as "none" (no files on asset), "folder" (multiple files on an asset), or single file extension (single file on asset and provides the extension, as before)
- File Uploads will go to a temporary S3 location on stage 1 while stage 2 upload completions performs checks, including for malicious file extensions or MIME types, before moving files into an asset for versioning
- File uploads restricted to 10 upload initializations (stage 1) per-user per-minute to minimize DDoS possibility and maximize system availability
- UploadFile now supports upload types for assetFiles and assetPreview to better support the separation of the uploads. This will allow for future enhancement support of adding filePreviews, separate from assets.
- Workflow execution final steps, which return files to an asset, are now rigged to use the new uploadFile lambda to support all file checks before versioning as part of an asset and to now support pipelines that return asset previews. This process follows an alternate external upload stage where presigned URLs are not needed due to the direct access nature of pipelines into the assets bucket (still uses temporary locations for security).
- AssetFiles API now brings back additional information for each file such as size, version, version created, and if the file is a versioned prefix folder or a file
- Support for empty asset creation and/or throughout life cycle of an asset (uploads no longer required during asset creation)
- Asset uploads in the UI now keep their original filenames and no longer change them to the asset name.
- The concept of "primary file" in an asset has been removed to support assets being truly multi-file
- New File URL Sharing action/modal for files in the file manager that generates presigned URLs for all files or folder selected
- Web The front-end asset upload has been heavily modified to support the new backend asset changes
- Now supports choosing multiple files and/or entire folders
- Files now keep their original names and are no longer changed to the assetName
- Supports the presigned URL and multi-stage API calls needed now for an upload (including support for splitting large files into multiple parts for parallel upload)
- Supports stage and file error recovery options, including proceeding with certain failed uploads that will be discarded
- Comments are no longer a supported field as part of upload, as this functionality has been moved to creating asset versions
- The assetFiles API now supports additional paths for functionality including
../fileInfo,../createFolder,../moveFile,../copyFile,../archiveFile,../unarchiveFile,../deleteFile,../deleteAssetPreview,../deleteAuxiliaryPreviewAssetFiles,../revertFileVersion. ListFiles now provides additional data about each file. - Web The front-end asset download for multiple files has been updated to support downloading an entire folder's worth of files in parallel
- Note: This still fetches individual files based on their presigned URL for automation, it does not pre-ZIP files on a server and may still cause issues if hundreds or thousands of files need to be downloaded
- Web The asset viewer file manager has been rewritten to support new features and richer user experience
- Instead of having a separate redundant icon view of files in the right pane of the file manager, it now shows file information such as file name, path, size, and any version information. For top-level asset nodes and image files, this will show the Preview file or actual file (image type files) now. This supports preview files now for both assets and files. See DeveloperGuide.md for documentation on preview file support (non-auxiliary).
- Added buttons for various downloads of files and folders
- Added ability and button to create sub-folders in an asset
- ViewAsset button still shown on files for asset 3D visualization, file-specific metadata, and file versioning
- Web Execute Workflow in View Assets now allows the user to choose which file on the asset will be processed due to the new multifile support implementation of assets
- Web Enhanced asset file management capabilities with comprehensive file operations:
- Added new API endpoints for file operations: fileInfo, moveFile, copyFile, archiveFile, unarchive, deleteFile, getVersion, getVersions, revertFileVersion
- Implemented file versioning with UI for showing files, knowing what version you are looking at, and reverting to a version
- Implemented file archiving which uses S3 delete markers (versus a permanent delete that removes the entire file)
- Added support for cross-asset file copying with proper permission validation (must stay within the same VAMS database)
- Implemented detailed file metadata retrieval including size, storage class, and version history
- Added permanent file deletion with safety confirmation to prevent accidental data loss
- Implemented proper error handling and validation for all file operations
- Asset files and versions will now show a flag for archived files and indicate if the asset is part of the current version files' version
- Web (Breaking Change) All new asset versioning capability and version comparisons
- Asset versions must now be manually created and will no longer auto-create when editing the asset or uploading files
- New APIs are defined for asset versioning for create, get, and revert options
- Asset table has changed fields and new asset version and asset file versions tables are created, which require a database migration...
v2.1.1
[2.1.1] (2025-01-17)
This hotfix version includes bug fixes related to dependency tools and library updates.
This release may require a installation of the latest aws-cdk library to either your global npm or as part of your local VAMS infra folder. Please re-run "npm install" in VAMS infra to install the latest local dependencies for existing deployments.
Bug Fixes
- Fixed and added Poetry export plugin library used during Lambda layer building due to Poetry no longer including "export" as part of the core library.
- Fixed Dockerfile container environment variable formats to no longer use the deprecated Docker format.
ENV KEY VALUE->ENV KEY=VALUE - Fixed 3D Metadata Labeling pipeline use-case to use the latest Blender version due to Alpine APK support deprecation for earlier specified versions.
- Fixed 3D Metadata Labeling pipeline use-case state machine Lambda to not hard-code the
us-east-1region for IAM role resource permission and use the stack-deployed region instead. - Updated aws-cdk dependency versions to the latest and updated GitHub CI/CD pipeline build checks
v2.1.0
What's Changed
[2.1.0] (2024-11-15)
This minor version includes changes to VAMS pipelines, use-case pipeline implementations, and v2.0 bug fixes.
Recommended Upgrade Path: A/B Stack Deployment with data migration using staging bucket configuration and upgrade migration scripts for DynamoDB tables in ./infra/upgradeMigrationScripts
⚠ BREAKING CHANGES
- Due to packaged library version upgrades in the solution, customer must make sure they are using the latest global installs of aws cli/CDK
- Pipelines are now changed to support a new pipelineType meaning, and the old pipelineType was renamed to pipelineExecutionType.
- Execution workflow input parameter names to pipelines have also changed, which can break existing workflows/pipelines.
Due to DynamoDB table structure changes, a A/B Stack deployment with migration script is recommended if there are existing pipelines that need to be automatically brought over.
Features
- Re-worked infrastructure CDK components and project directory structure to split out use-case pipelines (i.e., PotreeViewer/Visualizer Pipelines) from the rest of the lambda backend and stack infrastructures. This will allow for future upgrades that will split these components completely out into their own open-source project.
PotreeViewerPipeline(previously VisualizerPipeline) is now baselined to the new standard use-case pipeline pattern to support external state machine callbacks (i.e., from VAMS pipeline workflows)-
PreviewPotreeViewerPipeline(previously VisualizerPipeline) can now be registered and called from VAMS pipeline workflows (suggested to be called from a preview type pipeline) via the 'vamsExecutePreviewPcPotreeViewerPipeline' lambda function.
- Added a new use-case pipeline and configuration option for
GenAiMetadata3dLabelingPipelinethat can take in OBJ, FBX, GLB, USD, STL, PLY, DAE, and ABC files from an asset and use generative AI to analyze the file through 2D renders what keywords, tags, or other metadata the file should be associated with. Pipeline can be called by registering 'vamsExecuteGenAiMetadata3dLabelingPipeline' lambda function with VAMS pipelines / workflows. - Added a new use-case pipeline and configuration option for
Conversion3dBasicthat can convert between STL, OBJ, PLY, GLTF, GLB, 3MF, XAML, 3DXML, DAE, and XYZ file types. VAMS pipeline registrationoutputTypewill define for each pipeline registration what the output file extension type will be. -
- This pipeline for non-GovCloud deployments is enabled by default in the infrastructure configuration.
- Web Added
pipelineExecutionTypeto VAMS pipelines (previouslypipelineType) and added a new context topipelineType. Current pipeline types areStandardFileandPreviewFile. These are implemented to support future roadmap implementations of different pipeline types and auto-executions options on asset file uploads. - Web Added
inputParametersto pipelines to allow the optional specification of a JSON object which can be used within a pipeline execution to set pipeline configuration options. This is set at the time of creating a VAMS pipeline. - Added
inputMetadatato pipeline inputs which automatically pulls in asset name, description, tags, and all metadata fields of the asset to a pipeline execution. This can also be used in the future to pull through user-defined inputMetadata at the time of an execution with additional UI/UX. - Changed
inputPathandoutputPathof pipeline function execution inputs toinputS3AssetFilePathandoutputS3AssetFilesPath - Added
outputS3AssetPreviewPath,outputS3AssetMetadataPath, andinputOutputS3AssetAuxiliaryFilesPathpipeline execution parameter inputs to support different location paths for asset data outputs and writing to asset auxiliary temporary path locations - Added
outputTypefor user-specified expected file extension output for pipelines based on the VAMS pipeline registration. OutputType is not enforced and is something pipelines need to work into their own business logic as appropriate. -
- All asset write-back locations are now temporary job execution specific to allow for better security, file checks, proper back-versioning into an asset, and to start abstracting pipelines from writing directly to assets. Once the UploadV2 process is completed in a future update, direct access by use-case pipelines to S3 asset buckets will be removed in favor of API uploads / presigned URLs for storage abstraction.
- Updated
processWorkflowExecutionOutputlambda function (previouslyuploadAllAssets) to also account for metadata data object outputs of pipelines to update against assets. Preview image output logic is stubbed out but will not be fully implemented until the new upload / storage process overhaul is completed in a future version. - Added
credTokenTimeoutSecondsauthProvider config on the infrastructure side to allow manual specification of access, ID, and pre-signed URL tokenExpiration. Extending this can fix upload timeouts for larger files or slower connections. Auth refresh tokens timeouts are fixed to 24 hours currently. -
- Implements a new approach for s3ScopedAccess for upload that allows tokens up to 12 hours using AssumeRoleWithWebIdentity.
- Web Added PointCloud viewer and pipeline support for
.plyfile formats, moved from the 3D Mesh 3D Online Viewer - Web The asset file viewer now says
(primary)next to the assets main/primary associated file. The primary file is what get's used right now for pipeline ingestion when launching a workflow. - Changed access logs S3 bucket lifecycle policy to only remove logs after 90 days
- Added lifecycle polcies on asset and asset auxiliary bucket to remove incomplete upload parts after 14 days
Bug Fixes
- Fixed CreateWorkflow error seen in v2.0 (Mac/Linux builds) with updated library dependencies and setting a standardized docker platforms across the board to
linux/amd64 - Re-worked PreviewPotreeViewerPipeline (previously VisualizerPipeline) state machine and associated functions to properly handle errors
- Fixed benign logger errors in OpenSearch indexing lambda function (streams)
- Fixed existing functionality with
processWorkflowExecutionOutput(previouslyuploadAllAssets) not working - Fixed pipeline execution to properly account for asset file primary key names that contain spaces. Previously, could cause pipelines to error on execution.
- Web The asset file viewer now appropriately shows multiple files that are uploaded to the asset
- Web Hid the
View %AssetName% Metadatabutton for top-level root folder on asset details page file manager that led to a blank page. The metadata for this is already on the asset details page. - Fixed GovCloud deployments where v2 Lambda PreTokenGen for Cognito are not supported, reverted to v1 lambdas that only support Access Tokens (instead of both ID and Access token use for VAMS authorizers)
- Fixed GovCloud deployments for erronouesly including a GeoServices reference that is not supported in GovCloud partition
- Fixed KMS key IAM policy principals (for non-externally imported key setting) to include OpenSearch when using OpenSearch deployment configurations
- Added logic to look at other claims data if "vams:*" claims are not in the original JWT token. This is in prepartion for external IDP support and some edge case setups customers have.
- Fixed CDK deployment bug not deploying the required VPC endpoints during particular configurations of OpenSearch Provisioned, Not using all Lambda's behind VPCs, and using the option to use VPC endpoints
- Web Fixed bug where adding asset links had swapped the child/parent asset (WebUI only bug, API direct calls were not affected)
- Fixed CDK deployment bug of encrypting the WebAppLogsBucket when deploying with ALB and KMS encryption. The WebAppLogsBucket cannot be KMS encrypted when used for ALB logging output.
- Fixed bug for exceeding PolicyLimitSize of STS temporary role calls in S3ScopedAccess used during asset upload from the Web UI when KMS encryption is enabled.
- Increased CustomResource lambda timeouts for OpenSearch schema deployment that caused issues intermitently during GovCloud deployments
- Fixed bug in constraint service API that was saving constraints on POST/PUT properly but was erroring on generating a 200 response resulting in a 500 error
- Fixed bug in OpenSearch indexing (bad logging method) during certain edge cases that prevented adding new data to the index
- Fixed bug in CDK storageResource helper function where S3 buckets were not getting the proper resource policies applied
Chores
- VisualizerPipeline now re-named to PreviewPotreeViewerPipeline as the previous name was too generic and other "visualizer" or viewer pipelines may exist later
- 'visualizerAssets' S3 bucket renamed to 'assetAuxiliary'. This bucket will now be used for all pipeline or otherwise auto-generated files (previews/thumbnails) associated with assets that should not be versioned
- 'visualizerAssets/{proxy+}' API route and related function re-named to 'auxililaryPreviewAssets/stream/{proxy+}'. This function is used for retrieving auto-generated preview files that should be rapidly streamed such as the PreviewPotreeViewerPipeline files.
- Renamed and moved
uploadAllAssetslambda function handler. It is nowprocessWorkflowExecutionOutputand moved to theworkflowsbackend folder - Updated Workflow ListExecutions to write stopDate, startDate, and executionStatus back to DynamoDB table after an SFN fetch where the execution has stopped. This is done for performance / caching reasons.
- Workflow executions are now limited to only 1 active running...
v2.0.0
Highlights
- CDK Infrastructure Overhaul: This release represents a major overhaul of the CDK constructs, splitting the core logic into multiple nested stacks to support more scalable deployment configurations.
- Configuration System: A new CDK configuration system has been introduced using
config.jsonandcdk.jsonfiles. Many previously implemented features, such as OpenSearch or Location Services, can now be turned on or off. - New Configuration Options: Numerous new configuration options have been added, such as VPC/subnet management, Application Load Balancer (ALB) static web support instead of CloudFront, KMS encryption, OpenSearch configurations (including the ability to turn off OpenSearch), and more. These options can be toggled based on specific deployment requirements.
- Security Controls: A major aspect of this release focuses on security tightening and controls. Implementers will now be able to deploy across AWS partitions, including GovCloud, and have more control over WAF, FIPS, Lambdas in VPCs, and Docker SSL Proxy configurations.
- New Access Control System: A new Attribute-Based Access Control (ABAC) and Role-Based Access Control (RBAC) system has been implemented, replacing the previous Cognito group-based access control. This provides fine-grained access control to various VAMS resources.
- Asset Tagging and Linking: A new mechanism for adding tags and tag types to assets has been introduced, along with the ability to create parent/child and related-to links between assets within the same database.
- Image and PointCloud Viewers: Support for Image and PointCloud file visualizations has been added, including an infrastructure data pipeline to support viewer conversions for LAS, LAZ, and E57 input formats.
- Upgraded File Manager: The web assets viewer has a new file manager UI/UX for viewing asset files and provides functionality for uploading multiple asset files within folders.
- Email Subscription System: A new email subscription system has been implemented which allows VAMS users to subscribe to various data changes. Asset data objects are the first to be implemented as part of this version to allow users to receive notifications when new asset file versions are uploaded.
- Performance and Bug Fixes: Various performance improvements and bug fixes have been implemented, including API input validations, optimizations for OpenSearch indexing, log group naming, unique resource naming, and workflow execution handling.
- Deprecations and Removals: SageMaker pipeline types have been removed to focus development efforts on Lambda pipelines.
⚠️ BREAKING CHANGES
- Possible break CDK configuration and feature switch system using
./infra/config/config.jsonfile. Some backwards compatibility with existing CDK deployment commands. - CDK overhaul to split core logic into 10+ nested stacks means that an in-place upgrade for existing stack deployments is not possible, use A/B deployment.
- Lambdas converted into inline code functions with layers (away from Lambda ECR-backed containers).
- (SEO breakage) Switch Web infrastructure to use React hash router instead of web router to support ALB configuration option, which breaks search engine optimizations (SEO).
- New ABAC/RBAC systems will require new roles and constraints to be set up to allow application access. Existing Cognito groups will no longer be recognized, and user memberships must be transferred to the role and constraint mechanisms.
- SageMaker is no longer a pipeline type available. Existing SageMaker pipelines should be converted to be executed from a lambda pipeline.
- Restrict VAMS workflow pipelines to only have permission to lambdas that contain
vamsin the function name by default. If you have external pipeline lambdas, please add invoke permissions for them to the appropriate workflow execution role or update your lambda function name to containvams. - Pipelines created using the default lambda artifact sample will now need to be re-created and re-inserted into workflows due to using different database fields to store the name of these.
/assets/all(PUT) API call is deprecated in favor of using the existing/assets(PUT) and the newer/ingestAsset(POST) API.- Previously created workflows of pipelines that had pipeline nodes that didn't use
wait_for_callbackneed to be re-created/re-saved from the VAMS UI or modified in the AWS Console to removeTaskToken.$from node tasks parameters if there is no callback on that node. - API response bodies for data retrieval calls that return several records have been standardized to
responseBody: {message: {Items, NextToken}}.
Features
- Implement CDK configuration system using
./infra/config/config.jsonfile. - Implement local Docker package build file configuration override to support customization in
./infra/config/docker/Dockerfile-customDependencyBuildConfig(such as in cases of HTTPS SSL proxy certificate support). - Add default template files for various configuration environments (commercial (default- config.json), GovCloud).
- Implement new CDK environment system variables using
./infra/cdk.jsonfile. - Add global stack resource tagging.
- Add global new role permission boundary support.
- Add global new role name prefix tagging.
- Implement feature switch system and storage for Web feature toggling (new DynamoDB table).
- Web Load/cache enabledFeatures as part of the backend web configuration load to the frontend.
- Implement GovCloud feature switch which toggles other features on/off based on GovCloud service support and certain best practices.
- Implement FIPS support configuration option.
- Implement WAF configuration option (existing WAF functionality, ability to now toggle off).
- Implement Global VPC configuration option used for particular configuration needs.
- Support new VPC/Subnet generation.
- Support an option for external VPC/subnet imports (instead of new VPC generation).
- Added implementation of LoadContext Deployment configuration to support VPC context loading before main deployment.
- Support an option for auto-adding*new VPC endpoints based on other configuration switches (*with some exceptions in particular configurations that will still auto-add regardless of this flag).
- Support putting all deployed lambdas behind VPC (FedRamp best practices for GovCloud).
- Implement ALB configuration option for static WebApp delivery (replaces CloudFront when enabled).
- Requirement Note: ALB tied to a registered domain that must be provided.
- Support WAF (if used) to deploy globally or regionally based on ALB/CloudFront deployments.
- Support for using public private subnets for ALB.
- Support/Requirement for SSL/TLS ACM certificate import for ALB.
- Support for optional externally imported Route53 HostedZone updating for ALB deployment.
- Implement KMS CMK encryption configuration option for all*at-rest storage (*with some S3 bucket exceptions in particular configurations such as ALB use).
- Support new key generation on stack deploy.
- Support option for external CMK key import instead of new key generation.
- Disable all KMS CMK keys use implemented previously when configuration feature disabled (e.g., S3 bucket SNS notification queues). Uses default/AWS-managed encryption when KMS CMK disabled.
- Implement OpenSearch provisioned, serverless, or no (neither serverless nor provisioned enabled) open search configuration options; No open search will disable VAMS asset search functionality.
- Implement location service configuration option and feature switch (existing location service functionality, ability to now toggle off).
- Web Hides Map view from Assets web page when turned off.
- Implement point cloud visualization configuration option (existing pipeline functionality, ability to now toggle off through configuration file).
- Add VAMS upgrade migration scripts to support A/B deployments and data migration between stack deployments in
./infra/deploymentDataMigration. - (Future Full-Implementation) Implement authentication provider configuration option and feature switch. Note: Currently, only the Cognito
useSamlconfiguration flag is observed (moved fromsaml-config.tsfile), other auth types will cause an unimplemented error. - Implement new initial ABAC/RBAC access control systems to allow for fine-grained access to various VAMS resources (built on the Casbin open-source library).
- ABAC defines the primary constraints and access controls.
- ABAC currently supports resources of Databases, Assets, and "APIs".
- Note Databases and Assets control primary VAMS storage resources. APIs control access to top-level system functionality (administrative pages, pipelines/workflows, etc.).
- RBAC roles map to ABAC constraints to allow for backward compatibility with role/group-based access systems.
- ABAC constraints can also map directly to users if organizations choose to go solely with the ABAC system.
- Removed the previous Cognito group and constraint system.
- Note Starts to reduce dependency on Cognito functionalities.
- Created default admin role and constraint groups on new VAMS deployment. Stack deployment user will be auto-added to this new role group.
- All lambdas now check access against the new ABAC constraints system.
- Web Allowed Web routes controlled by ABAC constraints.
- Web Administrative UI pages to support roles, role membership, constraints, and constraint membership modifications.
- Implement new tag and tag type mechanism for adding additional information on assets (tags/tag types are currently global across all databases).
- Note Requirement that Tags must have a tag type assigned.
- Web Ability to search tags on assets on the asset search page.
- Web Ability to assign/unassign tags to assets on asset creat...
v1.4.0
What's Changed
- Metadata schemas for VAMS databases by @archieco in #86
- feat: Easily replace terms Asset and Database by @archieco in #88
- feat!: Support uploading folders as assets by @ravij3 in #92
- fix: repair regression on createPipeline by @archieco in #93
- fix: single folder single file upload by @ravij3 in #95
- fix: dependency conflict was causing downloads to fail by @ravij3 in #94
- Added File Level Metadata by @ravij3 in #96
- feat: hiding sign up by @ravij3 in #104
- OpenSearch Integration by @archieco in #103
- chore(deps): bump semver from 5.7.1 to 5.7.2 by @dependabot in #105
- fix: download asset only if they are marked as distributatble by @ravij3 in #106
- fix: repair first deployment with opensearch by @archieco in #107
- Rel 1.4 fixes by @ravij3 in #108
- fix: Release fixes by @ravij3 in #109
- Rel 1.4 fixes by @ravij3 in #113
- fix: simplify auth constraints screen by @archieco in #115
- Fixing download links and cancel button on update asset button by @ravij3 in #117
- feat(web): improvements to metadata component by @archieco in #110
- add multi file archiving capability by @amigitamz in #112
- chore(deps): bump certifi from 2022.12.7 to 2023.7.22 in /backend by @dependabot in #111
New Contributors
- @amigitamz made their first contribution in #112
Full Changelog: v1.3.1...v1.4.0
v1.3.1
What's Changed
- One small improvement to RBAC
- A fix was made to the workflow table on the ViewAsset screen
Full Changelog: v1.3.0...v1.3.1
For a full description of changes in the 1.3.0 release, see https://github.com/awslabs/visual-asset-management-system/releases/tag/v1.3.0
v1.3.0
What's New
- VAMS operators can now leverage SAML to federate users from their primary identity provider such as Microsoft Active Directory or any SAML IdP with Amazon Cognito.
- A new Role Based Access Control system enables VAMS administrators to provision access to users in accordance with their roles and responsibilities.
- A configuration preview of Attribute Based Access Control demonstrates fine grained access control to VAMS assets using metadata.
- New support for long running Pipelines with Step Functions' wait for callback feature. Pipeline execution can now take up to 1 year whereas before they were limited to 15 minutes for Lambda functions or 24 hours for Sagemaker Jobs.
- Additional automated security tests are executed on every change made to VAMS in its continuous integration pipeline leveraging the Automated Security Helper (ASH, https://github.com/aws-samples/automated-security-helper)
Detailed Changes
- chore(release): 1.2.0 by @ravij3 in #45
- feat(web): add new model visualizer supporting .obj, .gltf, .glb, .st… by @stephcurt in #42
- feat: apigw authorizer for amplify config endpoint by @archieco in #47
- Jjbain add security tests to ci script by @jjbainAWS in #52
- chore: prettier configuration and reformatting by @archieco in #51
- chore: made corrections to links in changelog by @archieco in #53
- eslint by @archieco in #54
- feat: enable cloudfront compression by @archieco in #48
- Federated authentication using SAML. by @archieco in #57
- added automated security helper by @jjbainAWS in #56
- Adjusted default permission for S3 buckets to match latest security changes by @AMZN-stankowi in #62
- Update README.md by @archieco in #61
- Issue 63 Fix - Windows Build Error by @copystart in #64
- Authrbac1 by @archieco in #60
- feat: Fine grained authorization rule definition by @archieco in #67
- fix: Hitting Execute Workflow button from the assets page doesn't work by @ravij3 in #70
- fix: automatically navigate to asset page once asset upload completes by @ravij3 in #71
- fix: resolves issue #68, workflow editor added extra pipelines by @archieco in #74
- fix: renaming userpool causes failures in existing stack by @ravij3 in #72
- fix: Updated cdk-nag suppression by @archieco in #77
- fix: congitoUsername --> cognitoUsername, added dependency to userGroupAttachment by @lmarbleAWS in #79
- docs: file formats list for online viewing by @archieco in #81
- fix: updated the workflow editor by @archieco in #80
- fix: cdk nag suppressions for python 3.9 and nodejs14.x by @archieco in #78
- feat: Support long running pipelines with Step Functions' wait for callback feature. by @archieco in #76
- chore(deps): bump requests from 2.30.0 to 2.31.0 in /backend by @dependabot in #82
- fix: resolve to fast-xml-parser 4.2.4 by @archieco in #89
- Resolved a couple of issues related to new authorization functionality. by @archieco in #85
- Jjbain update asset ux fix by @jjbainAWS in #83
New Contributors
- @AMZN-stankowi made their first contribution in #62
- @copystart made their first contribution in #64
- @lmarbleAWS made their first contribution in #79
Full Changelog: v1.2.0...v1.3.0
v1.2.0
Highlighted Changes
- A new asset upload wizard was created that enables users to upload files, create metadata in key/value pairs, and select workflows to execute on the asset upon upload.
- The asset detail screen now includes controls to update metadata for the asset.
- The user interface was updated to use the Cloudscape design system.
- Users with software that runs in containers or AWS Lambda functions can provide the ARN (Amazon Resource Name) referencing ECR image URI or AWS Lambda function to leverage their software to transform assets in VAMS.
- VAMS allow list of file types now includes STEP files.
- Assets are identified by a UUID while the user provided asset name is preserved with each asset rather than using the asset name as a natural key.
What's Changed
- fix: change log s3 bucket encryption type to S3_MANAGED by @stephcurt in #7
- fix: change all buckets to S3_MANAGED encryption by @archieco in #8
- feat(infra) parameterized region code by @jjbainAWS in #3
- chore(deps): bump certifi from 2022.9.24 to 2022.12.7 in /backend by @dependabot in #2
- chore: prettier configuration by @archieco in #12
- infra: adding uploadAssetWorkflow components by @ravij3 in #13
- ci: adding ci tools for backend repository by @ravij3 in #14
- fix: updating ci.yml by @ravij3 in #15
- ci: added ci for web and cdk by @ravij3 in #16
- chore(deps): bump axios from 0.21.1 to 0.26.0 in /web by @dependabot in #9
- fix: fixing loader-utils security vulnerability by @ravij3 in #18
- feat: Added uploadAssetWorkflow lambda function by @ravij3 in #20
- Metadata backend and frontend by @archieco in #19
- feat: updates to UploadAssetWorkflow stepFunction by @ravij3 in #22
- fix: downgrading default notebook platform by @ravij3 in #26
- chore(release): 1.0.1 by @archieco in #21
- feat: uploadAssetWorkflow stepfunction orchestration by @ravij3 in #27
- feat(web): awsui css replaced with cloudscape css by @archieco in #23
- chore: update broken links on DeveloperGuide by @ravij3 in #29
- feat(web): New asset upload screen by @archieco in #28
- chore: Repair copyright headers by @archieco in #30
- chore: update to list_objects_v2 by @archieco in #33
- fix: s3 copy_object calls include owner acct ids by @archieco in #32
- chore: remove unused resources by @archieco in #31
- chore(deps): bump werkzeug from 2.2.2 to 2.2.3 in /backend by @archieco in #34
- chore: adding fbx file formats for pipelines by @ravij3 in #35
- fix: security updates reported through automated security helper by @ravij3 in #36
- fix(web): update create asset buttons by @archieco in #40
- Several minor fixes and updates by @archieco in #37
- docs: Document the schemas of the dynamodb tables by @archieco in #41
- Jjbain bring your own arn by @jjbainAWS in #38
- docs: updates to documentation by @ravij3 in #43
- docs: update developer docs by @ravij3 in #44
New Contributors
- @stephcurt made their first contribution in #7
- @archieco made their first contribution in #8
- @jjbainAWS made their first contribution in #3
- @dependabot made their first contribution in #2
Full Changelog: https://github.com/awslabs/visual-asset-management-system/commits/v1.2.0