![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
I spent some time today watching the demonstration videos on Palantir's analysis blog.
For those who don't know, Palantir is a tool that is designed to help people look for patterns in large and complex sets of data. Currently it's mainly being used by the CIA and financial firms.
What they've built is really impressive. The first video in the blog, which shows how their government tool can analyze trends in public opinion about a new policy, takes a while to get going but is very fascinating toward the end. It's an example of the product being used for something outside its main application . Such use is a sign that Palantir has built something very exciting -- a general-purpose tool for analyzing and visualizing a wide variety of data.
The design problems inherent in creating such a product are interesting -- the creator of a new tool needs to work out a basic vocabulary of concepts and actions that is small enough to be learned easily but large enough to encompass a wide variety of desired behaviors. This sort of design problem is not trivial at all... it involves looking at a wide variety of use cases and coming up with a clear semantic structure that ties them all together.
This tool could be poised to take its place among the canon of software toolsets -- Photoshop for image manipulation, Excel for spreadsheets etc.
The trouble is, it's currently only being sold for large companies. The site has no pricing information, which is a sign that it's very expensive.
I would love to play with Palantir, but I have to admit that I'm not the average person. Here are some large datasets I think a lot of people would want to play with. There are existing tools for all of these areas, but they're often limited in terms of what you can do. I'm imagining bringing Palantir horsepower and mindset to problems like:
- Looking at how I spend my time. Of course, you'd need a well-marked-up schedule for your time spent in general, but time spent on the computer can be recorded automatically.
- Looking at how I spend my money. I haven't played with the various online money-management apps... perhaps they are better than I think.
- Finding out trends and thought leaders in new areas I'm excited about.
- Flight / Hotel / Car rental information. These sites are much better than they used to be, but they still don't let you slice and dice things to your exact preferences or look at multiple options without a lot of work. sidestep & kayak are currently best IMHO.
- Analyzing usage patterns of my flickr page, my blog, my youtube videos, and my other online presences. There are some analytic tools on the site already, but they are very limited in scope, limited on the visualization side, and are aimed at a small number of specific use cases.
- Looking at housing options. I want to be able to cull through a large number of listings and their offerings, look at proximity to various desirable (supermarkets I like, parks, etc) and undesirable (eg crime) options.
- Browsing options for a complex product purchase where the product has a lot of specs (eg cameras, projectors, cars)
If anyone knows good existing tools in these areas, let me know.
For those who don't know, Palantir is a tool that is designed to help people look for patterns in large and complex sets of data. Currently it's mainly being used by the CIA and financial firms.
What they've built is really impressive. The first video in the blog, which shows how their government tool can analyze trends in public opinion about a new policy, takes a while to get going but is very fascinating toward the end. It's an example of the product being used for something outside its main application . Such use is a sign that Palantir has built something very exciting -- a general-purpose tool for analyzing and visualizing a wide variety of data.
The design problems inherent in creating such a product are interesting -- the creator of a new tool needs to work out a basic vocabulary of concepts and actions that is small enough to be learned easily but large enough to encompass a wide variety of desired behaviors. This sort of design problem is not trivial at all... it involves looking at a wide variety of use cases and coming up with a clear semantic structure that ties them all together.
This tool could be poised to take its place among the canon of software toolsets -- Photoshop for image manipulation, Excel for spreadsheets etc.
The trouble is, it's currently only being sold for large companies. The site has no pricing information, which is a sign that it's very expensive.
I would love to play with Palantir, but I have to admit that I'm not the average person. Here are some large datasets I think a lot of people would want to play with. There are existing tools for all of these areas, but they're often limited in terms of what you can do. I'm imagining bringing Palantir horsepower and mindset to problems like:
- Looking at how I spend my time. Of course, you'd need a well-marked-up schedule for your time spent in general, but time spent on the computer can be recorded automatically.
- Looking at how I spend my money. I haven't played with the various online money-management apps... perhaps they are better than I think.
- Finding out trends and thought leaders in new areas I'm excited about.
- Flight / Hotel / Car rental information. These sites are much better than they used to be, but they still don't let you slice and dice things to your exact preferences or look at multiple options without a lot of work. sidestep & kayak are currently best IMHO.
- Analyzing usage patterns of my flickr page, my blog, my youtube videos, and my other online presences. There are some analytic tools on the site already, but they are very limited in scope, limited on the visualization side, and are aimed at a small number of specific use cases.
- Looking at housing options. I want to be able to cull through a large number of listings and their offerings, look at proximity to various desirable (supermarkets I like, parks, etc) and undesirable (eg crime) options.
- Browsing options for a complex product purchase where the product has a lot of specs (eg cameras, projectors, cars)
If anyone knows good existing tools in these areas, let me know.
