Rendered at 08:01:23 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Dramatize 1 hours ago [-]
One app that's using this technique (not exactly sure if it's the same) is Littlebird: https://littlebird.ai/
I also saw that HeyClicky started doing something similar but end up removing from the product.
zahlman 1 hours ago [-]
> reads the text of your focused window every few seconds through the Accessibility API
> It writes plain markdown
Where are the formatting decisions coming from?
Dramatize 24 minutes ago [-]
[flagged]
purplecats 3 hours ago [-]
neat. why not screenshot and tesseract (videos/images/viewport/etc)
Dramatize 2 hours ago [-]
Because you then have the macOS orange screen sharing warning/icon. I don't really want to record my screen, just the text is enough.
hetsaraiya 1 hours ago [-]
Agree
etchalon 2 hours ago [-]
I experimented with this exact same approach earlier this year.
It's barely sufficient, because, bluntly, most apps just aren't wired up right.
So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out.
Will try this app, to see if it improved on my own approach, but man, the hope levels are low.
albert_e 25 minutes ago [-]
I hope in near future _that_ layer of abstraction -- looking at a fairly standard application window with minor UI variations and reasoning about what area / labels within the UI mean what (possibly paired with app documentation) -- could probably become a light-weight fine-tuned vision model it itself that can run fully locally.
hetsaraiya 1 hours ago [-]
I haven't read about how the Codex Appshots work yet, but this can be used to extract text properly. I guess. How does this idea look to you?
Dramatize 21 minutes ago [-]
Appshots is fine for what it is, but not great for building ambient context about your daily usage/work.
Dramatize 2 hours ago [-]
Yeah don't get your hope up too much. I need to push through more cleaning up of what's captured. Let me know how it goes, keen for some feedback
zahlman 1 hours ago [-]
One of the reasons people like TUIs is because the text is always just right there.
I also saw that HeyClicky started doing something similar but end up removing from the product.
> It writes plain markdown
Where are the formatting decisions coming from?
It's barely sufficient, because, bluntly, most apps just aren't wired up right.
So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out.
Will try this app, to see if it improved on my own approach, but man, the hope levels are low.