WordPressParser asks WordPressClient to get blog-posts…

Is this really what a parser should do? Asking to get blog posts is not “parsing” by any definition of the term.

It’s like scissors sending you an invitation to come over to your local barber shop. Now, scissors surely have their business in making it happen, but triggering this entire chain of events doesn’t sound like something they should be in charge of.

For example, I need the media (photo and video) downloaded to disk. In
this case should I make ‘download_media()’ methods in the Parser
class, or is it better design to create a Downloader class?

I would create a separate class, because:

1. Downloading is a different responsibility than parsing – keep single responsibility principle in mind. It helps to keep track of which class does what as the project grows and allows to keep the code flexible – easy to change.

2. Different parsers may handle downloading in a similar, or even identical manner. By extracting this functionality into a separate class we can avoid code duplication, because the same Downloader implementation can be reused by different Parsers.

Should there be a class for BlogPosts? It seems to me that blog-posts
are objects (the equivalent to pages of a book, perhaps).

If they are nothing but strings – sheer text – it might not be necessary, but the cost of creating a BlogPost class isn’t all that high and at the very least it would make it explicit throughout your code that such and such variable represents a blog post and nothing else!

A good rule of thumb is to prefer domain specific types over primitive types. BlogPost means something. A piece of text could be just anything. We are only humans and using domain specific types helps us to write bug-free code, because we let the type system guard us against committing type mismatch offenses (eg. confusing a blog post’s url link with blog post’s content).

Disclaimer: I don’t know Python, I approach the question as a universal OOD question. If there is something Python-specific that escapes me here, I’d be happy if someone points it out.

I seem to be taking verbs, ‘download’, ‘parse’, and then converting them to nouns in order to make them appear as valid classes.

I feel very lost and confused. Again, the purpose for me is to learn how to properly make and design OO software. I’m not looking just to get the program to work.

The biggest challenge when starting out with object orientated design is the mental shift required to think about a problem in terms of different objects interacting to achieve a goal, rather than thinking of the procedural steps required to make the computer do something.

One thing that can help is an exercise in anthropomorphiation

Either grab a few of your friends/co-workers or grab a few toy figures or army men, go into a room and set about organizing the people to solve the problem when each person can do one thing and/or tell another object to do one thing.

You might start off with only a few people doing a lot of the work. Keep splitting up persons so that each person does just one thing while instructing other people to do other tasks for them.

Telling someone to do something is passing them a message, and message passing between objects is the core of OOD. You can also pass objects themselves to other objects, so imagine a person passing over a person to a third person if that third person needs this.

Try to design your system using this method. If a single person is doing too much, break them up into two more people. You will end up with a room over people saying out loud “I do this, then I tell Bob to do this”

You will notice that there is no code in this exercise. That is on purpose. The design of the system is far more important than the computer language used to implement it. Learning how to think in terms of objects as units of behaviour which achieve a goal through message passing among themselves is far more important than learning an OO programming language.

This excercise can help for both real world problems and also very abstract problems like the one you detail above because ultimately the details of how each person does what they do (opening a drawer to get a form or downloading the form form a HTTP web server) are not as relevant as what they say they can do.

It really depends on where you’re going. The reason to go with classes over method is classes are the starting point for some powerful OO patterns that you can take advantage of.

Take classes, add inheritance and polymorphism and now you can start taking advantage of contract based programming (i.e. Interfaces). This is very important. It allows you to define behavior abstractly and create clean lines of separation between the different parts of your code base. This is Inversion of Control (IoC). Your code should rely on abstractions/interfaces, not concrete classes.

Start programming against interfaces and you’ve decoupled the implementation of one piece of code from another. Now changing the code of one doesn’t have to affect the other, as long as the contract doesn’t change.

SOLID starts coming into play here, the Single Responsibility Principle (SRP) says that code should only have one reason to change. If your classes are separated cleanly that will be so.

Next the Open/Closed principle (OCP) states that your code should be open for extension and closed for modification. That means you can extend an interface, by making a new one that uses an old one, that way the old interface implementations are still valid. Similarly you can refactor what goes on in a concrete implementation of an interface as long as the input/output (our contract) stays the same.

Now that you have that separation you can do even more cool stuff like dependency injection and unit testing via mocked dependencies. With DI you structure your code so that all external dependencies are “injected” into the object via (typically) the constructor. The class should not do any service creation of it’s own and rely on it’s consumer to provide them. This is inline with the SRP. Doing this also gives you testability because now you can pass in mocked objects and isolate the testing to that class.

There are other things I could go into here, it’s a very broad topic. But that’s the high level. Basically, making your services into classes is a good idea if you think in any way that they could change or be done in a different way. For instance, your download, it should definitely be it’s own thing according the SRP. As to whether it should be a helper method or a class, ask yourself: Can I write this in a way that it will be consumed the same way by everything (input)? Are there other places you might want to download them to besides disk (output)? If you think there could be other cases, make it an interface with implementations that vary based on consumer. (You would retrieve that implementation with a factory).

Parser too benefits from being a class and/or interface because you can use polymorphism to vary the implementation at runtime based on what kind of thing you’re parsing as you’ve started to do.

What you’re missing here is that there should be one interface for Parser (or IParser by some typical standards). You want to have an IParser interface, a Parser base class that implements common code of all parsers, and concrete implementations of IParser that inherit from Parser, which would fulfill the contract in a site-specific manner. Then your main code path would use DI or a factory to get a concrete instance of IParser and do your work against that. Now it’s only dependency is on the interface and none of the implementations, which again, makes refactoring, testing and maintainability much better.

I could go on but I feel like I’ve rambled enough. Hope this is helpful.

I’ll offer some general problem-solving advice first then use it on your problem so you can see how someone with more experience might approach this.

Problem Domain

As you may have read elsewhere, your design and your code are a representation of your understanding of the problem you’re trying to solve. So, before you start coding and before you start designing, you need to analyze the problem itself; this helps make design of your solution easier to create. Along the way, if you can simplify the problem or your view of the problem, then that’ll make the solution simpler as well! And simple is good.

When I’m having trouble designing or coding something, a lot of the time it’s because I haven’t gotten a full understanding of the part of the problem I’m working on. Other times it’s because I’m not familiar with some design/implementation technique that would make things easier, so it’s good that you’re reading up on things!

When you’re analyzing a problem, look at who the entities of the problem domain are and how they interact: who does what to whom, who gets what from whom, what processes there are, where does data flow from/to, etc. If you can enumerate the entities and their interactions, this will help ensure that the solution you come up with is complete.

Design

When you design your program, you take the concepts from your problem space and try to map it into whatever language/paradigm you’re working in. In the case of OOP, this is objects and methods.

When designing for OOP, you should look at the nouns and verbs in your problem space: who does what to whom, who gets what from whom, etc. These will map into the objects (the “who(m)”) and methods (the “does”).

One thing which occurred to me while thinking about this is that it would be beneficial to consider which nouns are the subjects of an action and which are the objects — i.e. to distinguish the “who” vs “whom”. These should naturally lead to what objects perform which action on what. E.g.:

person.putOn(hat)

In this, the person is the one acting on the hat, so the method belongs to the person.

Although, sometimes a design might work out better if you reverse things:

hat.sitOn(person)

It depends on the rest of the design and how things map. These are things to look out for and consider.

Hard-and-fast OOP doesn’t always allow for things to be mapped from the problem space in the most succinct way; sometimes actions don’t seem like they belong to a particular object. In such cases, you could use just a plain function — otherwise you’ll get the noun-ification of verbs that you’ve noticed into classes. (Others have noticed this as well.)

Your Project

Let’s take the above and apply it to your project.

First we’ll look at the problem domain.

The aim of the software is to download images and video from my favourite blogs. (eg, Google Blogger blogs, or WordPress hosted blogs.)

What are the players? The actions? The processes? How are things related (in time, in location, etc.)?

Well, there are blogs, blogs have posts, and posts have media. Blogs are hosted by services. Services have APIs which let you access the blogs/posts/media* — the APIs allow you to download a blog (feed) in a particular format (JSON, XML etc.) and the blog’s content will be arranged in a specific structure in this format; part of this content will be references to media.

*I’m not sure if any of the APIs you will be using will be required to access media items of the service’s posts, or whether you can access them via regular web URLs, so we’ll err on the side of caution here and assume you need an API to access them.

Now let’s consider how data flows in the problem domain: You send a request to the API of the service and get a response with feed content, which includes references to media items. You extract those media references, then use the API to get a copy of the media items, then save them to disk in files. (I find it helpful to draw this sort of interaction out on paper, say with a sequence diagram.)

Next, let’s see how the problem domain maps to OOP (+ functions):

I said above that nouns in the problem domain will be mapped to objects in your program. This is true, but not necessarily everything will be mapped; some of the nouns in this problem space are at remote locations (i.e. the servers), so you might not need to model them in your program (but it’s handy to know they’re there).

Starting with the big items: On the remote side of things, there is the Service. The Service has an API. On our side of things, we have Us (the program), and something to talk to the Service’s API. So, the API connection — or client — is a concept that we need to implement, which you have done already.

Our client will be connecting to the API to download a blog feed, so the API client class (noun) should have a downloadFeed method (verb/action).

This is where things get a bit tricky design-wise, as you’ve found, because we have multiple levels of abstraction for essentially the same concept: there’s the abstract notion of a blog and its posts & media, but there is also the more concrete representation (JSON, XML) of the blog feed that the API client will receive from the service.

So, what should we do? Well, if the API client is the only thing that is going to deal with the feed, then we can probably just leave it in the concrete representation in each of the different API clients. But if we are going to have anything else in the program deal with the feed data, then we should use some common representation (e.g. a blog feed class, a blog post class, and a media item or media reference class) which all the API clients should return for rest of the program to use.

Should the feed data be used by other parts of the program? My answer is: yes. We can turn to a couple of Best Practices to see why: the Single Responsibility Principle (SRP), and Separation of Concerns (SoC).

SRP states that every class or method should be responsible for doing one thing, and one thing only. The purpose of our API clients is to communicate with the different APIs on behalf of the rest of the program. Let’s look at how keeping our data inside the API client class would fit in with this:

We download data from the API with the client and get a specific representation of the feed. This is ok, it’s what the client is supposed to do.

Next, we extract media information from the feed; this is something of a sub-responsibility in this case: the feed format is directly related to the API, but the API client is only for communicating with the API. In this case, extraction is an action so we’ll want to use a function, but it would be tolerable to have this function be part of the API client class. A helper class might be needed to help do the extraction, if extracting the data is complex enough — this is what you have constructed as your Parser type, which I’ll address in a little bit.

Then, the images have to be downloaded (an action, thus a method), which we decided earlier is in the realm of the API client, since it has to communicate with the API to do this.

Finally, the images have to be cached to disk (another action, though this one might need to be noun-ified). This is in violation of the responsibilities of our API client: the client is for communicating with the service API, not for caching things to disk — caching is a separate concern! Furthermore, having the clients do the caching would mean that the same logic would have to be repeated in every client, which is a violation of DRY.

So, it will be better to have the caching logic outside of the API client so that it can be used across clients. This means that we will need have to have an external, general representation of the feeds so that they can be consumed by the caching code.

Each API client then has to transform (verb) its specific representation (“whom” noun) of the feed to the general representation (another “whom” noun) for use outside the client. This should be done by a function (maybe with a helper class).

So far, our flow is as follows: The API client downloads an API-specific representation of the blog feed, it transforms this representation into a general representation, it then returns this representation to the outside program. The outside program takes this general representation and gives it to the caching logic, which caches it to disk.

This flow is good, except we’re missing one thing: the media! We still have to download the images and videos!

The media has to be downloaded at some point between when the blog feed is downloaded and when things are cached to disk — this is another design point: you should look at when things can be done and when they should be done.

One obvious point to download the media would be right after the data from the feed has been extracted and put into the general representation, but before it is returned from the API client to the rest of the program. The media could be added to the output of the feed function; so things would go something like this:

def downloadFeed():
    rawFeed = self.downloadRawFeed()
    commonFeed = self.extractFeedDataFrom(rawFeed)
    self.downloadAndPopulateImagesOf(commonFeed)
    return commonFeed

There is an alternative to this, and that is to have the media downloaded right before they’re cached — this way, you won’t end up with a feed that contains every single media item from that feed in memory at once!

What you could do is populate the general representation with, say, instances of a MediaReference class that holds the information about the media, along with a reference to the API client so that it can actually download it, then the caching code can tell the MediaReference instances to download their media to give to it, one at a time (so that they’re not in memory all at once).

And there you have some input on the design of this thing. (Damn, what a long post!)

I actually don’t know if I have made good choices here.

Your start was pretty good, so keep at it! You can do it!

Parser

A quick note on what you’re calling a parser.

As some noted in their comments, what you’re calling a parser is not an actual parser, though I’ve seen many people refer to such as a parser, so don’t feel bad about that. A real parser (see the computer science defn), along with a lexer, takes an input string (say, some JSON) and turns it into a tree of objects, a hierarchy. That’s it. It doesn’t do anything with that data after that point.

What you’re wanting to do is to scrape the data for information. A parser (either a JSON or XML parser, depending on what your APIs use) could be helpful for this, getting the API data into a form which is easily scrapable by your code so you don’t have to do a bunch of string manipulation.

Ok, I’m done now.