Dudd, I suspect we'd all like to help you, but do you really expect humans to read this?

Try formatting like this:

I bet good formatting would solve quite a few of your questions.

Haha, guess it does leave something to be desired. I'll blame it on posting at 2AM. Anyways, gaming_mouse managed to somehow decipher it, so props to him

Cross joins may be obnoxiously slow once you get a decent sized table. I'm not sure how many Jeopardy! questions you have, but my small research shows 6000 eps * 62 questions = 372,000 questions.

I'm not sure how SQLite deals with cross joins, but you could be accidentally creating a Cartesian product, which would be 372,000 * 372,000 rows (this won't ever complete, btw).

SQL Server (among other databases) do optimize appropriately with a WHERE clause, though: http://technet.microsoft.com/en-us/l...ql.105%29.aspx. I don't know how SQLite deal with it. I just want to warn you that if you decide to look at the whole result set, you won't ever be able to pull it up because you will be doing a Cartesian product.

The point of using FTS isn't speed per se. The point is that you gain many tools that you otherwise wouldn't have access to. These tools have the extra effect of allowing you to build sane, faster, and more flexible schemas.

I'm going to say it again, because maybe it wasn't clear that I was serious, but I have no idea why you would want to use SQL to solve this problem. Just seems unbelievably painful considering your entire dataset is so small and easily kept in memory.

Yes, I would love to be surrounded by others more technical than myself. Even though my boss has talked about taking on more analytical work and there is very rich data at my current healthcare company to do such a thing, ultimately I would have to build an entire analytics department from the ground up.

I actually think its a strong possibility that I would be given free reign to build projects on my own in 3 - 6 months once I completely automate all my reports and am building user friendly excel dashboards. But, this would be completely reinventing the wheel and I would be on my own. There's no point in doing this so I am preparing for a big move right now, which is outlined in the data science bootcamp thread I started.

I don't want to do a whole sales pitch, but we're looking to do a bit more service work where we work with companies like this to kick start their data work but with the medium term goal of transitioning everything back to their internal teams (so not being reliant on 3rd party contractors in perpetuity). PM me if interested.

for the jeopardy DB, why don't you just store the clues/responses in a Map<String, String> and iterate through that to find keywords? shouldn't be too slow, right? or is that too simplistic of an approach to that problem

How ugly is the following "patch" I just added to a website?
They have a div with the id "breadcrumbs" which is obviously meant for breadcrumbs but for some reason they don't want to use them. They do however want an extra header to appear in that location (it's basically just empty background) whenever a visitor is on the "blahblah" page or subpages of it.
[breadcrumbs div is not generated on any page by the template as I have turned it off]
Structure of the site is .../lang/top_level/... so it's .../en/blahblah or .../en/blahblah/subblah/ohnoesevendeeper etc.

I just append a div tag with id breadcrumb and the header text they want to the containing div (via jQuery) if the url contains the target-string

Edit: styling is done via CSS #breadcrumbs h2 { ... }

Shrug. Stuff like that should be done on the back end but whatever. I would use window.location.href over jquery I guess.

Yeah doing it on the backend > JS was my thought as well. But it's not big enough an issue that I'll spend time on it

IMO, marketing and data analysis is one area where you shouldn't worry about "reinventing the wheel."

The stock stuff is so horrible because few -- if not none -- of this software is made by rainmakers or people on the ground. The software makes a bunch of assumptions about your business that simply is not true in most cases. There is such a variety of businesses in the continuum that there is just no way at all that you can make something pre-packaged. Of course, this doesn't stop companies from selling their software, but I've never ever seen a software that fit the purpose of the company.

I think 3 to 6 months is plenty of time to create something, but at the same time, if you stupefy to the point that "anyone" can use it, you will end up creating something that is utterly worthless. In my opinion, the person who analyzes the data has to have full exposure to the raw data and the ability to creatively ask the data for information. There are so many variables and so many questions that can be asked, there is simply no way you can create something that resembles stock software.

IMO, skip the user-friendly dashboards, use raw code and SQL, and ask to hire people who are able to use the raw tools. PostgreSQL, Python w/ NumPy, PyLab, Pandas, and Excel is an incredibly powerful combination, and way better than any GUI dashboard I've ever seen.

Ok I have to ask because I avoid it like the plague but why would I use Excel if I have access to the raw data and skills in the other stuff you've listed?

Just to clobber it into something usable for less technical folks or are there any major advantages I'm missing?

I ask because I know very little about Excel/Calc and lots of work in ERP is gettng existing data in other formats into a sane format/into the DB of your software. It usually involves the data being held in Excel sheets of sorts. I remember a couple of projects where one of the first steps was always sucking Excel data into a DB.

IMO, there is immense value in looking at raw data. You can quickly re-sort, delete unwanted data, rebuild graphs, and many other things. For me, it is much faster and smoother than recompiling and creating charts.

Not to say that you can't live without it, but I'm saying I wouldn't be able to live without it.

No. I think the non-technical folk should see the final results in an understandable format. Yes, sometimes tabular data is good, but that depends on what you are trying to show. I sort of think of Excel as an intermediate presentation layer, except that it isn't always toward others.

Yep. The first step for me is extract from the prefab program.

This is the type of contracting work that we would be looking for in order to transition from basic financial reporting to inferential and predictive analytics. For example, we have medical encounter data from about 300k members and building a model to determine which members are most likely to have a high cost event (cancer, heart attack, diabetes) by looking at diagnosis code, lab results, cpt code, etc... could yield some very interesting results. We basically do nothing with the data except summarize it in excel pivot tables and bar charts with the occasional line chart thrown in there. I'm hoping I don't stick around too long but if for whatever reason I do, I will get in contact with you.

Thanks for the huge post as usual. I should clarify a bit - I have ~ 10 reports that are very straightforward, month-end type of reports that are going to be fairly static and don't have as much potential for a more detailed analysis. These are the 'user-friendly' types of reports that I can automate with VBA and set up a dashboard that saves the company enormous amounts of time. My bosses were very impressed by a simple conditional formatting formula so they are absolutely blown away when I show them these excel products.

I think you are right on that our data is so specific and so incredibly vast (1000's of tables that took me months to feel comfortable knowing the field names) that an external pre-packaged program would not be useful. Re-inveting the wheel was meant as stand in for me having to learn the tools all alone and implement the analysis without having an expert by my side to hold my hand. Lots of useless frustrations before the magic happened. I have a book on pandas/NumPy but at this pace it would be a few years before I would be in the driver's seat putting it to use. Its possible that I could convince the department to hire an expert or contract out someone like jj but it seems like a pretty convoluted path for me to make it to the next step.

I'd rather move to the talent than convince the bosses to bring the talent to me. I'm a role player not Lebron (not yet...).

Tonight's tough lesson: If you are using a VM, you may want to delete the old snapshots as you create new ones, preferably before you have over 200 snapshots.

daveT sorry, I don't quite get your answer. What do you usually use Excel for? For me it's always been the data source pretty much (so basically my involvement with it ended after I moved the data from Excel to somewhere else). I've always thought it might be useful to actually use Excel for other stuff but I really don't know how/what I should look at etc.

Anyone have experience hiring psd to css freelancers? My partners and I are just going to outsource it since we really just need code and we already have the designs (sooner than later). Looks like we could get quality work in the $25/hour range on elance... but I've never done anything like this before.

It'd also be awesome if we could get someone to help style our phonegap app as well, but I'm not sure what to expect from someone in terms of knowing how to setup a development environment to test it (using ionic framework).

Anyone have any thoughts?

It's just a nice tool to use to clean and munge, both before and after data storage.

Think of this: suppose you have a sales history and you want to create a chart. Would it be easier to pile down and delete what you don't need, resort, move things around, and create charts on the fly in a spreadsheet, or would it be easier to rewrite queries, test, write new code, make a new chart (which is more code), and reiterate? Excel would let you do this on-the-fly analysis in a few minutes. Personally, I prefer to keep my mind focused on one particular task at a time. I don't want to think about *how* to extract an answer with code if I can just pull the answer out and consider other perspectives on the raw data.

There are limitations to databases and limits to Excel. They seem to counteract each others' limitations effectively.

i think that's a narrative that's applicable to you or some specific people know, and you're generalizing from that. it's one of those stories that sounds good and makes sense, and for which you can find anecdotal examples. but the fact is that most poker players i've met would not make good programmers, and vice versa. in both cases, it's usually not a matter of lacking the raw ability -- it's that becoming good in either of those things has so much more to do with inclination, natural interest, and motivation than it does with any specific kind of intelligence, even if that intelligence is a requirement.

I want to learn stuff involving languages for web-based stuff, such as Ruby/PHP/SQL, etc. What's a solid IDE for me to test out random sets of code? I've tried downloading Visual Basic and ran into issues (plus it was HUGE). Any help would be appreciated.

To be clear, I just want to be able to type in code and run it to see what it does. Like a simple printf("Hello world\n"); would produce it within the program.

Edit: I'm downloading XAMPP to install on my PC. This will let me execute PHP/MySQL type stuff locally, correct? I've also downloaded Sublime Text to help me edit the code. Good so far?

I doubt you'll need more than sublime text and XAMPP, should be good. When you do, you'll probably want to move onto linux (or linux in a virtualbox). There is probably better, for for me I do webdev with a terminal window, a code-highlighting text editor (KDE Kate usually for me) and Google Chrome.

edit: might be nice to add postgresql once you're happy with the basics, depends what you're doing I guess.

What's the difference between Linux and me using windows? XAMPP and Sublime Text is working fine for me, so far, although I kind of prefer DreamWeaver since it alerts me to which lines of code are causing issues.

once deployed to a real live server out on the internet, it's very likely to be a linux. it helps if your development environment is quite close to the production setup. Apache, PHP etc all will these days run on Windows, but it's not their natural home, some things may behave in a different manner. Silly things like a case-insensitive file system can trip you up later down the line. I doubt it really matters at all for now, though. I'm not even sure it matters at all these days tbh, I moved my apache/php etc dev to linux a long long time ago.

Wouldn't I just need to use a Windows based server then? I've heard about lowercase/uppercase causing issues when making the switch since certain servers read them differently. Any other examples of issues?

But thanks for the heads up. It's something I'll consider when I advance beyond echo, hehe.

hi guys, i m pretty new to programming...by that i mean, i just started trying about 20 minutes ago.

so i downloaded python 3.4.0 and one of the steps that all the tutorials have (i've checked a few different ones) is where you type something like:

>>> for i in range(10):
... print i
... print "bye"

and it will have

0
bye
1
bye
2
bye

etc etc...but i always get this syntax error even though i could swear i'm doing the same thing they are in the tutorials:



what did i do wrong lol (i didn't try to put the proper spacing in the italicized example but i did the indent/4 spaces in the screenshot)

thanks