Indeed you should use a One2Many from Author to books.
Once you have this field you can add it to the author modules tree view, but currently the One2Man widget on tree view only shows the number of records in the relation (but not the name).
If you wan to show the name you can use a Function field of char types. For example:
book_names = fields.Function(fields.Char("Books"), 'get_book_names')
def get_book_names(self, name):
return ', '.join(b.rec_name for b in self.books)
Then if you add the book_names field on the list view you will see the list of books separated by a comma.
Normaly we use a relate action to open the books associated to an author.
Indeed it should be possible to create a nested tree view using probably an UnionMixin to show the name of the author and the name of all its books as children records. But then you will be only able to show the name of both models and it requires to create a custom module to join both of them.