spencer's cafe

Search

SearchSearch

persistent data

Jan 10, 2024, 6 min read

Log

  • 2023-11-09
    • thinking about what the best way to publish these kinds of things is. If you have the infrastructure, you can get easy feeds of data via API and a way to append to them, but actually customizing how it looks is a whole other ball game
      • there’s htmx that aims for you to just write html to do all this and handle functionality (kinda ugly though)
      • closer to user-side is mmm.page which might get close to achieving this stuff with mmm.town which i imagine gives support for dynamic data
      • maybe for now i assume this is solved because mmm.page fills this niche and i work on the infrastructure part
    • just discovered the concept of microformats, which seems to try to introduce concept of shared common schemas (recipe, review, etc.)
    • read paper from SIGCHI on end-users publishing structured data on the web which notes self-describing formats (JSON) as much more agentic than schema-based formats (CSV, Sheets)
    • what if i just made a simple JSON format
      • Designed for prototyping (when you are rapidly changing schema) but also for maintaining small datasets, serialized to JSON so it loads entirely into memory, maybe can XPath to get different segments dynamically and the server handles loading parts of it?
      • open API access so you can sync into your preferred interfaces
      • simple web / mobile web interface for editing
      • bring your own storage provider (simple instructions for S3, cloudflare, google drive, icloud, paid hosted sync version)
      • how would this look? it probably looks different from a table.. in some ways because of the self-describing nature, maybe a tree? maybe both? or a table view that allows you to actually add new properties easily / have that be encouraged?
      • some interface for converting into SQL / object / etc. schema? idk if this is really useful or necessary tho, but without it why would you start with this / how do you actually make it easy to convert? the main thing is that the APIs need to be the same or very similar to make it simple to switch over
        • generate typescript interfaces and JSON-LD schema

Summary

okay maybe this should just be simplified to be:

  • an easy, cheap, accessible, low-maintenance way to give people who know the tiniest bit of web dev access to a simple persistent database. Closest metaphor would be a persistent and synced local storage-esque interface. Ideally hook into an existing cloud provider people have.
  • automatic RSS feed (and then also a reader?)
  • plaintext should be able to just store in your obsidian store, needs direct manipulation
  • becomes collaborative so you can make personal collections but then also create shared libraries
    • see all the links recently shared with raymond
  • ACL levels:
    • private
    • open to all with link
    • discoverable

Direction

something like a tuple store x json x rss x spreadsheet(? this is the optional interface)

  • no authentication, this is for small apps where you just want to build something quick and have it persist and persist to an optional persistor. Is this basically tinybase? Should definitely try using tinybase.
  • how to deal with joins well?
  • how to deal with rich objects well?
  • What would it look like to compile from plaintext into data formats? (sql or datalog)
    • Compiling is magical though, it is not quite the vision of whole data

Context

Persistent data refers to data that saves permanently and is not erased when you reboot an app. This is perceived as difficult when working in non-local contexts because it usually means you need a database, and that usually refers to a lot of overhead (hosting it, deploying it, securing it, etc.) and not the small, fun, easy kind that is embedded like sqlite, which is just a fancy file format with a library for working with that file format.

The easiest way to persist data is using a simple file. You can use a text file and just append new lines for each new row of data. You could upgrade to a more structured file format like JSON for getting some sample schema or something like XML for even more.

The trouble with this is once you want to host a website or a server on the internet and get these files to persist. It’s hard to get real machines to power anything on the internet these days, like the ones you probably type on. You usually “rent” computing space on large company servers, which don’t include persistent file storage.

My goal database for homecooked apps I want to be able to use a simple embedded database (like sqlite) with modern severless, compute-on-demand infrastructure. This seems to require some sort of simple, “persistence as a service” for embedded / small databases, that fulfills a simpler need of the traditional hosted database services. This database should be local-first and support syncing with multiple peers simultaneously writing to the same place (how does this work with sqlite which is optimized for a single writer?). It should be small and require low maintenance and be “magically” performant for simple use cases.

Embedded Databases

see https://twitter.com/jaredforsyth/status/1217649814793572352 for more

Resources

  • fly.io: offers “volumes” which are persistent storage for fly apps, up to 3gb storage on free plan
    • very cool LiteFS in beta, which is embedded sqlite that automatically replicates without needing a managed db service
  • cloudflare d1
  • Google Cloud Run
    • ~$200 per month for persistent file storage on the “starter” plan
  • glitch with Pro ($8/month) can “boost” 5 apps that let them always be on (and they always have persistent file storage)
  • repl.it has persistent file storage like glitch for $2 a day (actually you can make this free)
  • digital ocean droplet scaling slider starting at $4 / month, a lot of manual config needed though probably

References

Graph View

Backlinks