Re-consider design choices of KVS local storage client

[Pijukatel]

Current KVS storage client implementation follows several design choices that are not compatible together:

• Key name can have a dot in it - for example: someKey.json
• Locally, the values are stored in a files with that have {keyName}{optionalSuffix}
• Files without suffix are accepted by the storage client

This leads to several edge cases where the local client will behave differently than the platform client and to situations, where the local client will have to guess as the same situation can be interpreted in different ways.

Example edge cases:

1. On the file system, there is a pre-existing file someKey.bin
   Possible interpretations:

• It is a key: someKey and its value is of type bytes
• It is a key: someKey.bin and its value is of type bytes
  The local client has to guess and pick one of those options

2. Saving keys with names that look like filename with suffix can lead in missing keys.
   This will create 2 keys on platform,

const kvs = await Actor.openKeyValueStore('keyTest', {forceCloud: true});
await kvs.setValue('INPUT', { key: 'value'});
await kvs.setValue('INPUT.json', { key: 'value2'});

,but same will create only 1 key locally:

const kvs = await Actor.openKeyValueStore('keyTest');
await kvs.setValue('INPUT', { key: 'value'});
await kvs.setValue('INPUT.json', { key: 'value2'});

Benefits of using suffixes in the storage client:

• DX when working manually with the files

Benefits of allowing files without a suffix:

• Backwards compatibility
• ?

[barjin]

I believe this issue is more fitting for https://github.com/apify/crawlee , as that's where the memory storage implementation of KVS lives (see here).

---

To add to the topic - we shouldn't conflate the KVS key and the file name in this implementation. How about storing the data as files with extensions (for better debugging DX), but using a "directory" file mapping the keys to the filenames?

Or better yet, should we just use the keys as the file names verbatim? Modern OSes should use file signatures to determine the file contents anyway.

re: breaking change - both the solutions above would definitely be breaking, making this a Crawlee v4 candidate

[janbuchar]

Related - apify/crawlee#3189