It's time to put web designing and website development in its place. We
often think of web sites as monumental feats and large constructs which are
difficult to build, maintain, and proliferate. Additionally, they are large
degrees of programming that take an army to keep up. While it is true that some are like that, the majority are just simple directories that
your web server (be it IIS or Apache) delivers to the public. Graphic designers
and web hosting services would like you to believe that web presence is
something monumental. Visit any web designer site and you will see all the
fluff words and words-of-cajole that tell you how they will make your business
better. Take what they say with large grains of salt. I now offer you a paradigm shift to counter their point-of-view. You
can think of the following items as its equivalents:
If we think of things this way, it is actually more close to the truth.
These are just some evidence that to a lot of greedy people, virtualness can
equate to a-lot-of-bullshit-ness.
That is essentially the main impediment pitting the Linux and Windows political camps. For human beings, we just want something we need done for us. It would be nice if it was done nice and neatly, but ultimately, it needs to be done. That is what Windows does best: it formalizes the things that we do most: word processing, excel, email, code editing.... and presents it in a very nice way. Things it doesn't do well are the odd little things we need once in a while. For example: find all the files that I have referenced "John and Mary" in this directory. In Unix, it's grep -i "john and mary" * . That is the forte and the distilled credo of unix. Provide basic simple tools and have the tools work well. This article: Migrating to Linux not easy for Windows Users is quite apt. In order to use the hammer, saw, or even digital volt meter (DVM), you need to have some sort of basis or background. The basis for Windows users do not prepare them for Unix. The expection then is for Unix to look at least 90% like Windows. The folks in the Linux camp are making their OS go in that direction.... whether they will succeed is anyone's guess.
Being aligned with your work as a programmer or software engineer. It is a good article about acceptance.
Web-scraping may have several meanings so I will just define it in the context of this article so as not to confuse you, the reader. It is to take some specific information available when you visit a specific webpage, and do something with it. Things that are intesting to a person when you go to a website may be:
Well, if they are interesting to a person, maybe they would be interesting to a program too. As an example, go to this URL: http://finance.yahoo.com/q?s=msft&d=v1
The returned page has lots of information available regarding the Microsoft stock. The specific information that your application needs depend upon you. It can be a piece of relevant text (current stock price, volume information, amount changed), a list of links with the latest press releases about Microsoft, or anything else interesting returned by the Yahoo server. Assuming that we are interested in the stock price, various methods of text processing can be used to extract the number that we are interested in. From that number, we can go on to more fully utilize our computer to perform many other calculations and presentations on our desktop.
Of course, these companies presenting the information intend for the public to access the information by hand (i.e. go to the site with your web browser, surf to the page, access the information as well as all the advertisement that accompany it). This is all fine and dandy if all we want is a handful of data (count your fingers)... but what if we want a basket-ful? Then what about the occational.... carload-ful? That's a job for the computer...
Ok, the typical pundit would say that this is the perfect example for using "web-services". I have touched upon web services a little a year ago. True, indeed, if the company exports that specific interface for that specific data that resides on that specific webpage. As of this writing, I still haven't found enough useful web-services to comment on. However there currently exists many sites that provide useful enough information not to ignore. Let me just rattle off a few besides the Yahoo finance:
In truth, the services provided by websites exists in their human accessible form (HaF). The "web services" proponent push for their computer accessible form (CaF). My contention is that the time-gap in order to have access to the HaF is too great to ignore. By the time CaF is made available, opportunities may have disappeared. Web Scraping is the alternative that can exist in the meantime. It creates an intermediary CaF from an existing HaF. I admit that the algorithm to extract the data is fragile and is broken once someone changes format of the webpage output. But if you can accept this drawback, you can make the web work for you today. Not tomorrow, next week, or next year.... now.
As a developer using the Python language, I find the ASPN Python Cookbook to be a valuable resource for information about a variety of things that you can do. The site mainly contain text recipes about how to perform certain tasks using Python. It is decently designed so that getting around isn't too much of a problem. Unfortunately, using the site IS very much a struggle. It is so slow and when using it, I can barely tolerate just a few clicks. Such a pattern of "difficult to use" sites are becoming common. Another example is Zope.org , which ranks right up there in terms of containing useful information.
I wonder what could be the culprit for the sites being slow? Is it because the application servers being used are too resource intensive that it takes longer to respond to a page request? Is it because the servers being run are too under-powered because the owners cannot afford a better server? Whatever the reason, it is a shame because there are enough users that patronize these two sites that make it worth while to give them a better option. Has anyone thought about mirroring these sites? This is a useful (relatively low-tech) pattern that is increasingly being used on the internet. It is one of the many ways which edge networks can be implemented in a low-cost manner.
These are some good tools I have recently come across:
I have been looking for a Python Class Browser for some time but have not come across a decent one. Despite all the bashing of Microsoft technologies, it must be admitted that Visual Studio is the premier tool of choice for developers. I find myself entirely dependent and hooked on its integration of a class browser when doing development with MFC. Why is a class browser such a powerful tool? Well, it allows the developer to abstract his thinking beyond the line-by-line code. In an object oriented world, we will assume the lower dependencies work. Just like we assume a nail and hammer works when we are trying to put together a cabinet or something even bigger (a house).
So once it is understood that our dependencies work, we just put it aside and concentrate on the problem we are concerned with. The class browser allows us to collapse the class and work with those classes which we are concerned with. Additionally, the class list display is not constrained by only displaying classes within one file. It displays the entire class hierarchy of a particular project. I know, some of you may say "what's the big deal about manually openning up a different file". Not much, actually. But it is an impediment in the thought process. If you do it enough time, all those little impediments add up to be a big stumbling block of creating your solution.
If anyone has been using such a tool for Python, please drop me a tip.