a5huynh/spyglass

A personal search engine that indexes what you want, exposing it to you in a...

A personal search engine that indexes what you want, exposing it to you in a simple & fast interface

Spyglass

<br />
<br />
    Download now:
    <a href="https://github.com/a5huynh/spyglass/releases/download/v2022.8.11/Spyglass_22.8.11_x64.dmg">
        <strong>macOS</strong>
    </a> |
    <a href="https://github.com/a5huynh/spyglass/releases/download/v2022.8.11/Spyglass_22.8.11_x64_en-US.msi">
        <strong>Windows</strong>
    </a> |
    <a href="https://github.com/a5huynh/spyglass/releases/download/v2022.8.11/spyglass_22.8.11_amd64.AppImage">
        <strong>Linux (AppImage)</strong>
    </a>
<br />
<br />
<a href="https://docs.spyglass.fyi">
    <strong>Documentation</strong>
</a> |
<a href="https://docs.spyglass.fyi/usage/index.html">
    <strong>Using Spyglass</strong>
</a> |
<a href="https://docs.spyglass.fyi/usage/lenses/index.html">
    <strong>Lenses</strong>
</a>
<br />
<br />
<img src="https://github.com/a5huynh/spyglass/actions/workflows/rust.yml/badge.svg">
<a href="https://discord.gg/663wPVBSTB"><img src="https://img.shields.io/badge/Discord-Join%20Now-blue"></a>



Introduction

Spyglass lives on your device crawling & indexing websites you want with a basic set of rules.

Web pages when condensed down to text are surprisingly small. With todays' incredibly fast CPUs and ample amounts of of disk space, you can easily create a personal library of wikis, blog posts, etc. that can be referenced instantly. Cut through the SEO spam of the internet by building your own index.

For users who have been frustrated with the current state of search and the internet, Spyglass offers a powerful solution to find exactly what you want.

See Launching & Using Spyglass to get started.

Traditional web search sucks

The short answer is that Google search results are clearly dying. The long answer is that most of the web has become too inauthentic to trust.

  • https://dkb.io/post/google-search-is-dying

Spyglass is a solution to the following common issues when searching the web:

  • Do you add terms such as reddit or wiki to your searches to narrow it down?
  • Do you get frustrated with overzealous autocorrect on your search terms?
  • Do you get frustrated with the terrible search some wikis/sites offer?
  • Do you scroll past dozens of SEO spam pages to find the recipe/review/blog post you were looking for?
  • Do you skip over a full-page of ads before getting to your actual search results?
  • Do you have private websites / data / documents that you'd like to search through?

How does it know what to crawl?

Spyglass expands on the ideas outlined in this paper by the Brave Search Team. There are currently a simple set of rules that will point Spyglass at a website and crawl only what you want. When available, crawling is bootstrapped w/ data from the Internet Archive to not overwhelm smaller websites.

Not all websites & not all data can be crawled by Spyglass. If you have something that you'd like to index and would like some help, feel free to ping me on our Discord server!

See Community Lenses to install lenses others in the community have built.

See Building your own lens to see how easy it is to build your own lens. Please share w/ the community when you're done!

Developer Guide

If you'd like to help, reach out on our Discord server to see what is currently being developed and how you can help usher in a new, better search.

See Building from source to get started building & contributing to Spyglass.

Issues

Collection of the latest Issues

a5huynh

a5huynh

enhancement
Comment Icon0

Lens building right now is black magic. Would be great to have a UI/validator of some sort to help folks build things out.

redraskal

redraskal

enhancement
Comment Icon0

Spyglass can easily hit archive.org rate limits.

Ideas:

  • Fallback to Google cache or live site
  • Swap between archive.org, live site, & maybe Google cache to avoid rate limiting and balance out the request load
skewballfox

skewballfox

bug
Comment Icon9

not sure which is the culprit so I added both. I'm assuming that since spyglass app is trying to register a shortcut, this has to do with the window manager I am using. I'm guessing you are trying to communicate with some settings daemon which doesn't exist.

upon launching the app image, an grey X window appears with the text "search" and this text displays to stdout:

afterwords the window crashes, the commandline seems to hang, no system tray icon appears.

about my system: OS: fedora 36 kernel: 5.18.13-200.fc36.x86_64 window manager: sway 1.7 app image version: spyglass_22.7.4_amd64.AppImage

sloganking

sloganking

bug
Comment Icon0

Depending on multi-display orientation, the search box will not be rendered on screen. This is with spyglass_22.7.4_amd64.AppImage on Manjaro Linux, using the Cinnamon desktop environment. So the search box appears to be horizontally aligned with the primary display, but not vertically.

Examples

dataO1

dataO1

development
Comment Icon3

Hi, im trying to make a nixos derivation for spyglass, but the build fails. Here are the error logs:

I have the following packages installed as buildInputs:

sbeckeriv

sbeckeriv

bug
Comment Icon3

Dearest maintainer,

I think caching calls in memory(?) to robots.txt for an hour would save a lot of calls to robots.txt when importing. I was going to say per session but i think a time limit would cover a lot of calls.

Thanks for your consideration Becker

pknessness

pknessness

enhancement
Comment Icon0

I love the crawl status screen. I think it might be nice to show, for each set of URLs, which lens is responsible for bringing that URL set into the queue.

I have a number of URLs which, I don't know how they got into the queue, so It would be nice to figure out which lens they're from so I can fix the lens. image

pknessness

pknessness

bug
Comment Icon2

I have an SMG lense, and searching in it is odd

When I search, for example, good egg, it doesn't show good egg galaxy image

However, when I search something like pirahna, it does image

It makes sense that searching pirahna shows good egg galaxy as thats where you fight the boss peewee pirahna, but it doesn't make sense that it doesn't show the page for peewee pirahna, which i do have indexed image

Is there some way in which I can better tune how spyglass goes through the indexed pages?

a5huynh

a5huynh

bug
Comment Icon1
  • Global shortcut sometimes doesn't work on Linux.
  • When it does work, search input is not focused.
  • Closing the app sometimes stalls (potentially an IPC issue?).
teovoinea

teovoinea

enhancement
Comment Icon12

Hey! A few years ago I set out to make a similar project but that was focused on local file search.

You can find it here: https://github.com/teovoinea/podium

Much of the architecture of spyglass seems to be very similar to podium. They both use tantivy for indexing and tokio for async. It seems like you've made some great progress on the UI and bundling it as an actual app. Much farther along than I got in that respect.

I haven't had the time to work on that project outside of maintaining the dependencies.

Would you be interested in 'absorbing' my local file indexers?

I still think there's huge value in building a tool like this. Especially one that can run entirely locally on your machine but still bring some AI smarts like object detection an OCR that's typically cloud only.

Versions

Find the latest versions by id

v2022.8.11 - Aug 26, 2022

See the assets to download this version and install.

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.10...v2022.8.11

v2022.8.10 - Aug 25, 2022

See the assets to download this version and install

🧙‍♂️ New startup wizard!

Screen Shot 2022-08-25 at 3 25 52 PM

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.9...v2022.8.10

v2022.8.9 - Aug 18, 2022

See the assets to download this version and install.

Bugfixes

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.8...v2022.8.9

v2022.8.8 - Aug 12, 2022

See the assets to download this version and install.

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.7...v2022.8.8

v2022.8.7 - Aug 10, 2022

See the assets to download this version and install.

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.6...v2022.8.7

v2022.8.6 - Aug 10, 2022

See the assets to download this version and install.

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.5...v2022.8.6

v2022.8.5 - Aug 09, 2022

See the assets to download this version and install.

Hotfix for local file search support

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.4...v2022.8.5

v2022.8.4 - Aug 09, 2022

See the assets to download this version and install.

Bugfixes

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.3...v2022.8.4

v2022.8.3 - Aug 06, 2022

See the assets to download this version and install.

This is a test build meant to test out the new local file search functionality, use at your own risk!

Bugfixes

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.2...v2022.8.3

v2022.8.2 - Aug 02, 2022

See the assets to download this version and install.

🐛 🔥 Hotfixes

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.8.1...v2022.8.2

v2022.8.1 - Aug 02, 2022

See the assets to download this version and install.

🐛 Bugfixes

What's Changed

New Contributors

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.7.4...v2022.8.1

v2022.7.4 - Jul 26, 2022

See the assets to download this version and install.

Bugfixes / Stability Improvements

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.7.3...v2022.7.4

v2022.7.3 - Jul 24, 2022

See the assets to download this version and install.

Bugfixes

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.7.2...v2022.7.3

v2022.7.2 - Jul 23, 2022

See the assets to download this version and install.

Plugins!

Screen Shot 2022-07-23 at 1 18 50 PM

Bugfixes

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.7.1...v2022.7.2

v2022.7.1 - Jul 06, 2022

See the assets to download this version and install.

🐛 Bugfixes

✨ What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.6.5...v2022.7.1

v2022.6.5 - Jun 29, 2022

See the assets to download this version and install.

✨ What's Changed

🐛 Bugfixes

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.6.4...v2022.6.5

v2022.6.4 - Jun 25, 2022

See the assets to download this version and install.

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.6.3...v2022.6.4

v2022.6.3 - Jun 22, 2022

See the assets to download this version and install.

🐛 Bugfix

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.6.2...v2022.6.3

v2022.6.2 - Jun 22, 2022

See the assets to download this version and install.

🐛 Bugfixes

✨ What's Changed

🧹 Chores

New Contributors

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.6.1...v2022.6.2

v2022.6.1 - Jun 03, 2022

See the assets to download this version and install.

What's Changed

lens-manager

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.5.28...v2022.6.1

v2022.5.28 - May 27, 2022

See the assets to download this version and install.

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.5.27...v2022.5.28

v2022.5.27 - May 26, 2022

See the assets to download this version and install.

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.5.26...v2022.5.27

v2022.5.26 - May 26, 2022

See the assets to download this version and install.

Some screenshots of new stuff:

Crawl status window, see whats taking up your queue/index!

Screen Shot 2022-05-26 at 11 11 35 AM

Cleaned up search results / delete button

Screen Shot 2022-05-26 at 11 14 19 AM

What's Changed

Bugfixes

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.5.15...v2022.5.26

v2022.5.15 - May 16, 2022

See the assets to download this version and install.

What's Changed

  • Fixed an issue w/ index writing because we were calling commit too often, move commits to a secondary task which also speeds up indexing as a secondary result.
  • By default, Spyglass will now only crawl URLs that have been specified in lenses. This setting is configurable if you want the crawler to act like a typical web crawler w/ the crawl_external_links setting.
  • The data_directory setting was added so users can change where the index + metadata is stored.
  • Better handling of crawler crashes, the crawler will now attempt to restart gracefully if something went wrong.

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.5.11...v2022.5.15

v2022.5.11 - May 11, 2022

See the assets to download this version and install.

⚠️ Backwards in-compatible release ⚠️

  • If you have existing lenses / settings these can be moved over to the new preferences folder.
    • On macOS this has changed from $HOME/Library/Application Support/com.athlabs.carto -> $HOME/Library/Application Support/com.athlabs.spyglass
    • On linux this has changed from $HOME/.local/share/carto -> $HOME/.local/share/spyglass
    • On Windows this has change from $HOME\AppData\Roaming\athlabs\carto -> $HOME\AppData\Roaming\athlabs\spyglass
  • Index schema has changed to support intra-domain indexing, so all indexes will need to be rebuilt
    • In consolation, the new bootstrapping feature makes this a lot faster 🚀

What's Changed

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.5.4...v2022.5.11

v2022.5.4 - May 04, 2022

See the assets to download this version and install.

Whats Changed

New Contributors

Full Changelog: https://github.com/a5huynh/spyglass/compare/v2022.4.30...v2022.5.4

v2022.4.30 - May 01, 2022

See the assets to download this version and install.

What's Changed

  • Closed #12 which means Windows users can now join in on the fun!
  • Better focus hand-off when the search bar pops into view.

app-v2022.4.29 - Apr 30, 2022

See the assets to download this version and install.

What's Changed

  • Fixing startup crash with large limits (#18)
  • Fix cursor moving during selection + focus on show (#19)
  • Force focus on search box when it pops into view

app-v2022.4.27 - Apr 28, 2022

⚠️ Removing the Windows build for now, since it's being held up by #12

This version moves over to the YYYY.MM.DD versioning system since it's easier to see when something was last released and semantic versioning doesn't quite work for a customer facing application.

  • Adds #9, a configuration setting for the shortcut.

app-v0.1.2 - Apr 27, 2022

⚠️ There's currently an issue w/ the Windows build that causes the UI to hang when searching

  • Fixes #3: Instant crash when hitting "Show lenses folder" or "Show settings folder" on non macOS distros
  • Fixes #6: Remove deprecated keys from tauri.conf.json due to tauri-cli update.
  • Fixes an issue where the tray menu wasn't correctly updated on non macOS distros

Information - Updated Sep 05, 2022

Stars: 812
Forks: 11
Issues: 14

Repositories & Extras

Misc

248

A CLI tool to easily get a new project up and running by using pre-made...

A rust cross platform rust boilerplate template to get up and running quickly

A CLI tool to easily get a new project up and running by using pre-made...

HAL : Hyper Adaptive Learning

Rust based Cross-GPU Machine Learning

HAL : Hyper Adaptive Learning

A crossplatform Rust bindings for the soloud audio engine library

Supported formats: wav, mp3, ogg, flac

A crossplatform Rust bindings for the soloud audio engine library

Rust crate providing cross-platform information about the notebook batteries

battery provides a cross-platform unified API to a notebook batteries state

Rust crate providing cross-platform information about the notebook batteries

A cross platform Rust library for efficiently walking a directory recursively

Comes with support for following symbolic links, controlling the number of

A cross platform Rust library for efficiently walking a directory recursively

debug-here: a cross platform rust debugger hook

Debuggers are a great way to examine the state of a program

debug-here: a cross platform rust debugger hook

Cross-Platform Rust Toolkit for Games 🏹

Android-related tools for building and developing application

Cross-Platform Rust Toolkit for Games 🏹

crossfont is a cross-platform Rust library for loading fonts and rasterizing

glyphs, using native font engines whenever possible

crossfont is a cross-platform Rust library for loading fonts and rasterizing

Yet another (x)fetch tool, written in rust with cross-platform in mind

This program was written specifically with cross-platform libraries, so in theory,

Yet another (x)fetch tool, written in rust with cross-platform in mind

Crossclip is a cross-platform Rust library for interfacing with the system

It has been tested on GNU/Linux, FreeBSD, Windows and Mac OSX

Crossclip is a cross-platform Rust library for interfacing with the system
Facebook Instagram Twitter GitHub Dribbble
Privacy