Skip to content

feat: Add AWS DynamoDB KV Storage tool#5111

Merged
HenryHengZJ merged 3 commits into
FlowiseAI:mainfrom
anatolyburtsev:feat/kv-storage
Sep 11, 2025
Merged

feat: Add AWS DynamoDB KV Storage tool#5111
HenryHengZJ merged 3 commits into
FlowiseAI:mainfrom
anatolyburtsev:feat/kv-storage

Conversation

@anatolyburtsev

@anatolyburtsev anatolyburtsev commented Aug 20, 2025

Copy link
Copy Markdown
Contributor

Summary

This PR adds a Key-Value Storage tool to Flowise, enabling persistent storage and retrieval of data with built-in versioning capabilities. It uses AWS DynamoDB as a backbone.

Tool has 2 modes: store or retrieval.

  • Store mode - stored object by key. Example response:
    Successfully stored value with key "total_comp" at 2025-08-20T05:41:09.608Z
  • Retrieval mode - return object by key with timestamp when it was created. Consumes optional argument nth-latest, e.g. value 2 returns second last version of the object. Example response:
{
  "value": "200",
  "timestamp": "2025-08-20T05:41:09.608Z"
}

Note

Retrieval mode works great with JSONPathExtractor introduced here

Example use-case

Monitor a website for changes by storing snapshots and comparing different versions over time. This is useful for tracking competitor updates, detect when news articles are updated or corrected, or track feature announcements and product updates. Diff together with old and new versions can be feed directly to LLM to make sense of it.

Implementation Details

  • Added shared AWS utilities in /packages/components/src/awsToolsUtils.ts for credential management
  • DynamoDB's has to have Partition key pk and Search key sk. Corresponding validation is done during configuration phase. The schema is
    • pk: The actual key name
    • sk: Timestamp as string (for sorting)
    • value: The actual data
    • timestamp: Human-readable ISO timestamp

Configuration

The tool requires:

  • AWS Credentials (via credential manager)
  • AWS Region selection
  • DynamoDB table name.

Testing

  • Added unit tests
  • Manual testing with actual AWS DynamoDB instances

Example of the flow:
Screenshot 2025-08-19 at 10 42 20 PM

Screenshot 2025-08-19 at 10 42 32 PM

Store mode node configuration:
Screenshot 2025-08-20 at 10 45 30 AM

Retrieval mode node configuration:
Screenshot 2025-08-20 at 10 45 40 AM

- Add AWS DynamoDB key-value storage tool for persistent data storage
- Add utility functions for AWS tools configuration
@anatolyburtsev anatolyburtsev changed the title feat: Add AWS DynamoDB KV Storage tool WIP feat: Add AWS DynamoDB KV Storage tool Aug 20, 2025
@anatolyburtsev anatolyburtsev marked this pull request as draft August 20, 2025 06:24
@anatolyburtsev anatolyburtsev marked this pull request as ready for review August 20, 2025 06:46
@anatolyburtsev anatolyburtsev changed the title WIP feat: Add AWS DynamoDB KV Storage tool feat: Add AWS DynamoDB KV Storage tool Aug 20, 2025
Comment thread packages/components/src/awsToolsUtils.ts

@HenryHengZJ HenryHengZJ left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you @anatolyburtsev !

@HenryHengZJ HenryHengZJ merged commit 736c2b1 into FlowiseAI:main Sep 11, 2025
2 checks passed
erhhung pushed a commit to erhhung/flowise that referenced this pull request Oct 5, 2025
* feat: Add AWS DynamoDB KV Storage tool

- Add AWS DynamoDB key-value storage tool for persistent data storage
- Add utility functions for AWS tools configuration

* update SNS tool

---------

Co-authored-by: Henry <hzj94@hotmail.com>
@OrhanA-Innodare

Copy link
Copy Markdown

Dear Anatolyburtsev,
Thank you for developing the AWS DynamoDB tool for Flowise. I understand that it can only store the attributes one by one and when asked for Retrieve gives only the latest one. But if you need to get the latest attributes for a specific user, if I have more than one user, I could not find anyway of doing it. I might have 100 users, and I want to get the latest attributes for a user. It may be that, in the data base there are 50 other users which loaded data on the table before the specific user that I want to get the data for. Is there any way of doing this in flowise or is that tool for a limited use only? Thank you in advance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants